relevance / diametric Goto Github PK
View Code? Open in Web Editor NEWDiametric is a library for building schemas, queries, and transactions for Datomic from Ruby objects.
License: MIT License
Diametric is a library for building schemas, queries, and transactions for Datomic from Ruby objects.
License: MIT License
Currently, Persistence::Peer uses edn-ruby library to convert Ruby object to edn. This is a good idea to keep compatibility over Persistence::REST. However, it loses flexibility to cope with any data type of datomic in Ruby saide.
JRuby can directly create datomic objects using Java API. Eliminating edn-ruby conversion from Persistence::Peer will be good for supporting more data types effectively.
Some are probably implemented, however, needs to make sure. Also, needs to document what queries are correspond to what.
Right now if I do something like:
t = Terrarium.new
t.turtles << Turtle.new(name: "Speedy")
t.save
the turtle is not also saved with the terrarium... I have to create and save the turtle first.
It would be much better if saves were transitive to all new objects, and they were all part of the same datomic transaction.
irb(main):001:0> require 'diametric'
NameError: Undefined method == for class 'ActiveSupport::BasicObject'
from org/jruby/RubyModule.java:2330:in `undef_method'
Everything's fine with JRuby 1.7.10.
It is easy to create Rails app, but nice to have sample app.
(I have a pullup to fix this; I'm creating the issue so I can tie it to that pull request).
When an association between a parent and a child, all objects save correctly. However if you then load the parent, traverse to the child (causing it to load), the child is marked as dirty. If you then save the parent, a second copy of the child is saved.
Hi,
I'm using Peer Service to connect to the Datomic.
I created Storage Service based on DynamoDB on Amazon by following https://www.youtube.com/watch?v=wG5grJP3jKY tutorial (using CloudFormation service, it's also referenced here http://docs.datomic.com/aws.html).
The tutorial ouptus following URI:
uri = "datomic:ddb://us-east-1/your-system-name/test-db?aws_access_key_id=<MY_ACCESS_KEY>&aws_secret_key=<MY_SECRET_KEY>";
Unfortunately, there's no way for me to put that into URI that connects to the database:
Diametric::Persistence.establish_base_connection({:uri=>"datomic:ddb://us-east-1/your-system-name/test-db?aws_access_key_id=<MY_ACCESS_KEY>&aws_secret_key=<MY_SECRET_KEY>"})
I'm being given
RuntimeError: :db.error/unsupported-protocol Unsupported protocol :ddb
If I change "ddb" into "free" I'm being given
RuntimeError: :db.error/invalid-db-uri Invalid database URI datomic:free://us-east-1/your-system-name/test-db
What would an approach to connect from Diametric directly to remote transactor that's exposed through setting up Datomic by Amazon Cloudformation service? Where do I find that URI?
Regards!
Hello,
I have following models:
class Badge
include Diametric::Entity
include Diametric::Persistence::Peer
attribute :category, Ref
attribute :date, DateTime
attribute :description, String
attribute :enum_value, String
attribute :enumeration, String, :cardinality => :many
attribute :id, UUID
attribute :name, String
attribute :type, Ref
attribute :value, Double
attribute :week, Integer
attribute :year, Integer
end
and
class Person
include Diametric::Entity
include Diametric::Persistence::Peer
attribute :name, String
attribute :id, UUID, :unique => :identity
attribute :badges, Ref, :cardinality => :many
attribute :address, String
attribute :city, String
attribute :state, String
attribute :country, String
attribute :zip_code, String
attribute :phone, String
attribute :email, String
attribute :age, Integer
attribute :gender, Ref
end
My intention is to create Person with two Badges.
badge = Badge.new(name: "average_weekly_revenue", value: 30.0, date: DateTime.strptime('2014-10-21T04:05:06', '%Y-%m-%dT%H:%M:%S'))
badge.save
badge2 = Badge.new(name: "some_other_badge", value: 20.0, date: DateTime.strptime('2014-10-21T04:05:06', '%Y-%m-%dT%H:%M:%S'))
badge2.save
so when I create new person:
user_id = UUID.generate
person = Person.new(name: "Natalia Jolie", id: user_id, badges: [badge, badge2])
person.save
I'm being given:
RuntimeError: java.lang.IllegalArgumentException: :db.error/not-an-attribute 17592186045435 is not an attribute.
get at diametric/DiametricListenableFuture.java:44
save at /Users/michalsiemionczyk/.rvm/gems/jruby-1.7.15/gems/diametric-0.1.3-java/lib/diametric/persistence/peer.rb:15
testD at /Users/michalsiemionczyk/Projects/yeti-backend/michals-test/test.rb:115
And that points to the
person = Person.new(name: "Natalia Jolie", id: user_id, badges: [badge, badge2])
line. The 17592186045435
number is a dbid of the first badge.
I've tried another approach described in the tutorial in here https://github.com/relevance/diametric#association
user_id = UUID.generate
Person.new(name: "Brad Dipp", id: user_id).save
person = Diametric::Query.new(Person, @conn, true).where(name: "Brad Dipp").first
person.update_attributes(badges: [badge, badge2])
person = Diametric::Query.new(Person, @conn, true).where(name: "Brad Dipp").first
puts person.badges.collect(&:name)
and that throws the same error that points to the person.update_attributes(badges: [badge, badge2])
line.
When I tried the tutorial it throws the same error on me.update_attributes(kids: [mario, luigi])
.
Naturally I'm fine when adding associations with :cardinality => :many, but for my schema I need multiple references.
How to make it work?
When following the wiki, https://github.com/relevance/diametric/wiki/Datomic-Pro-Version-Support, bundle install fails with:
16:37:24 INFO NaetherTransfer - Downloading: file:/Users/sbuxton/.m2/repository/com/datomic/datomic-pro/0.8.4122/datomic-pro-0.8.4122.pom
16:37:24 INFO NaetherTransfer - Downloading: http://repo1.maven.org/maven2/com/datomic/datomic-pro/0.8.4122/datomic-pro-0.8.4122.pom
16:37:24 INFO NaetherTransfer - Downloading: http://clojars.org/repo/com/datomic/datomic-pro/0.8.4122/datomic-pro-0.8.4122.pom
16:37:24 INFO NaetherTransfer - Downloading: https://repository.jboss.org/nexus/content/groups/public/com/datomic/datomic-pro/0.8.4122/datomic-pro-0.8.4122.pom
16:37:25 INFO NaetherTransfer - Downloading: http://files.couchbase.com/maven2/com/datomic/datomic-pro/0.8.4122/datomic-pro-0.8.4122.pom
16:37:25 WARN NaetherRepository - Missing artifact descriptor for com.datomic:datomic-pro:jar:0.8.4122
It appears the datomic pro jar file is no longer hosted in any of the maven repositories.
Currently, REST service is left behind. All tests pass so far, however REST also should have more queries or attribute types and so on.
Hello,
since I'm posting a lot you can expect me to become Diametric master and Diametric well documented library with nice FAQ ;)
I've been struggling with passing more than one argument to Diametric::Persistence::Peer.q
method.
Let's say I have a following code:
qString = "
[:find ?e
:in $ [?badge_name ...]
:where
[?e :badge/name ?badge_name]
]
"
It's crucial here that I use collection [?t ...]
(I need to create simple OR statetent and retrieve all he Badges that has a name that exist in the set specified by me).
I'm calling it through, eg.
result = Diametric::Persistence::Peer.q(qString, @conn.db, ['facebook_is_fan', 'facebook_engagement_gold', 'average_weekly_revenue'])
Works as expected. However, if I add additional source argument into my query string:
qString = "
[:find ?e
:in $ ?badge_value [?badge_name ...]
:where
[?e :badge/value ?badge_value]
[?e :badge/name ?badge_name]
]
"
I cannot come up with a way of adding additional parameter as source argument.
For instance,
result = Diametric::Persistence::Peer.q(qString, @conn.db, [[30.42], ['facebook_is_fan', 'facebook_engagement_gold', 'average_weekly_revenue']])
Returns
RuntimeError: Datomic Exception: :db.error/too-few-inputs Query expected 3 inputs but received 2
q at diametric/DiametricPeer.java:295
Isn't it caused by implementation of q()
method that contains the call for Diametric::Persistence::Peer.q(query, db, args.flatten(1))
and the call for Clojure APi should not expect only flattened list of source aguments?
I'd like to achieve sth like the suggested response for this issue https://groups.google.com/forum/?hl=pl#!topic/datomic/QZJrpbp-t-8
Thanks in advanve!
@yokolet and @rkneufeld : you two are doing a great job of working on this gem. If it is ok with you, I am going to change the owner to the Relevance account. Once I do that, could you add me as a collaborator on the new repo?
I would like to filter out for the objects that have , let's say value attribute.
How to perform "lower than A" AND "greater than B" query?
Using query = Diametric::Query.new(Badge, nil, true).filter(:>, :value, 15).filter(:<, :value, 40)
won't work, as the query.data.inspect
returns
[[:find, #<EDN::Type::Symbol:0x5d27af65 @symbol=:"?e">, :in, #<EDN::Type::Symbol:0x586c3faa @symbol=:"$">, [#<EDN::Type::Symbol:0x2811f8b1 @symbol=:"?valuevalue">, #<EDN::Type::Symbol:0x33de4104 @symbol=:"?valuevalue">], :where, [#<EDN::Type::Symbol:0x4f20b45b @symbol=:"?e">, :"badge/value", #<EDN::Type::Symbol:0x6c20eaf2 @symbol=:"?value">], [(> ?value ?valuevalue)], [(< ?value ?valuevalue)]], [15, 40]]
so in the end both the lower/greater than signs are applied to the same ?valuevalue
.
Is there any way I can enforce diametric to take 2 additional arguments (apart from writing a custom query in the string?)
Regards,
Michal
The Getting Started page of the wiki could use some love. Improve this documentation, and when done link README.md to this page.
Refactor Diametric::Persistence such that it becomes the primary module to include and the connection determines different behavior.
A suggestion from @yokolet:
.connect should take an argument :rest or :peer, #peer? and #rest? methods should be on the connection object
For example, given the entity definition and instances below:
class Person
include Diametric::Entity
include Diametric::Persistence
attribute :name, String, :cardinality => :one, :doc => "A person's name"
attribute :spouse, Ref, :cardinality => :one, :doc => "A person's spouse"
end
Person.create_schema
alice = Person.new(:name => "Alice")
alice.save
bob = Person.new(:name => "Bob", :spouse => alice)
bob.save
alice.spouse= bob
alice.save
currently, there's no easy way to find entity by "a person whose spouse's name is Alice" since spouse method returns dbid.
Currently, diametric Peer API supports a part of datomic API. It should support fully.
Datomic has :db.type/ref type to associate entities. Current Diametric attribute doesn't have the type as well as implementation. So, I'll start working on this.
My idea of refrence type is ...
Definition:
class Person
include Diametric::Entity
attribute :name, String, :cardinality => :one, :doc => "A person's name"
attribute :parent, Ref, :cardinality => :many, :doc => "A person's parent"
end
Person.schema
# Datomic transaction
# [{:db/id #db/id[:db.part/db]
# :db/ident :person/name
# :db/valueType :db.type/string
# :db/cardinality :db.cardinality/one
# :db/doc "A person's name"
# :db.install/_attribute :db.part/db}
#
# {:db/id #db/id[:db.part/db]
# :db/ident :person/parent
# :db/valueType :db.type/ref
# :db/cardinality :db.cardinality/many
# :db/doc "A person's parent"
# :db.install/_attribute :db.part/db}
Usage:
alice = Person.new(:name => "Alice")
bob = Person.new(:name => "Bob", :parent => alice)
chris = Person.new(:name => "Chris", :parent => alice)
Does this make sense?
RubyGems.org doesn't report a license for your gem. This is because it is not specified in the gemspec of your last release.
via e.g.
spec.license = 'MIT'
# or
spec.licenses = ['MIT', 'GPL-2']
Including a license in your gemspec is an easy way for rubygems.org and other tools to check how your gem is licensed. As you can imagine, scanning your repository for a LICENSE file or parsing the README, and then attempting to identify the license or licenses is much more difficult and more error prone. So, even for projects that already specify a license, including a license in your gemspec is a good practice. See, for example, how rubygems.org uses the gemspec to display the rails gem license.
There is even a License Finder gem to help companies/individuals ensure all gems they use meet their licensing needs. This tool depends on license information being available in the gemspec. This is an important enough issue that even Bundler now generates gems with a default 'MIT' license.
I hope you'll consider specifying a license in your gemspec. If not, please just close the issue with a nice message. In either case, I'll follow up. Thanks for your time!
Appendix:
If you need help choosing a license (sorry, I haven't checked your readme or looked for a license file), GitHub has created a license picker tool. Code without a license specified defaults to 'All rights reserved'-- denying others all rights to use of the code.
Here's a list of the license names I've found and their frequencies
p.s. In case you're wondering how I found you and why I made this issue, it's because I'm collecting stats on gems (I was originally looking for download data) and decided to collect license metadata,too, and make issues for gemspecs not specifying a license as a public service :). See the previous link or my blog post about this project for more information.
I think this line should be @data.delete_if { |e| e.to_i == entity.dbid }
I can submit a PR if you agree.
And thanks for the gem. It's great for learning datomic in a familiar setting.
Lets say I have an entity, Terrarium, and another entity, Turtle
Terrarium has an attribute defined as:
attribute :turtles, Ref, cardinality: :many
If I get the terrarium from the database with Terrarium.all, Terrarium.first, or Terrarium.where, all works exactly as expected (thank you!) but if I get the terrarium with find given its id, as in:
t = Terrarium.find 17592186045422
then the terrarium that is returned doesn't have its turtles properly reified... t.turtles will be a Java Set filled with instances of Java::DatomicQuery::EntityMap
I'll spend some time looking at this this weekend, but any hints would be welcome. I was surprised to see find defined in Entity while all the other finder-like things defined in Peer...
I wanted to start a conversation about cardinality/many.
I have a number of thoughts about meshing Ruby/ActiveRecord idioms with Datomics implementation. The most pressing is Datomic's usage of sets (if I'm not mistaken) as opposed to ActiveRecord heavily utilizing ordered and non-unique arrays.
I think it is probably sanest to stay as close to actual datastore implementation as possible. So I'll probably approach this via Set. This may be a little jarring for less Clojure-y developers coming to Datomic, so I think it might be worthwhile to keep the API users informed by complaining early (e.g. you try to assign an array as a default instead of a set).
Next up is retraction/protraction scheme. I think if the set-based approach is used this becomes a simple matter of producing a list of retractions and protractions based upon the difference between original and present values.
REST and Peer raise different exceptions, but both raise an exception when :db.type/uri is specified.
The hard part is to keep track tempids of referencees and referencers.
This implementation can be done by the third argument of tempid function, or use db/unique option to keep track.
In any case, Diametric needs some sort of meta (or helper) class in term of clarity. The save
method should save a single entity to datomic.
Hi, sorry if this seems a bit pedantic, but on the project you say you use the MIT license, but then the actual license doesn't look like the MIT license and you have a link to another completely separate license. This is project really released under the MIT license?
Connecting via Peer doesn't seem to support pro storage:
uri = "datomic:ddb://us-east-1/<dynamo-db-name>/<db>?aws_access_key_id=<key>&aws_secret_key=<secret>"
Diametric::Persistence::Peer.create_database(uri)
RuntimeError: Datomic Error: :protocol/unsupported Unsupported protocol :ddb
from diametric/DiametricPeer.java:83:in `create_database'
and
Diametric::Persistence::Peer.connect(uri)
RuntimeError: Failed to create connection
from diametric/DiametricPeer.java:71:in `connect'
Whilst I can connect via the datomic shell using the same uri without issue:
datomic % conn = Peer.connect(uri);
<{:db-id "sampledb-cdda2a0d-3952-49c9-bfb6-adf581d520d7", :unsent-updates-queue 0, :pending-txes 0, :next-t 1000, :basis-t 62, :index-rev 0, :index-root {:rev 0, :key "5293bcf5-617c-438d-a204-6c09d5f6c5a6"}, :log-root {:rev 1, :key "5293bcf9-c604-484c-aed4-2ed33c698074"}}>
Am I missing something here or are there limitations on the diametric Peer library?
A few basic features I want to add:
config/diametric.yml
is present)diametric:install
should exist, so when i say: rails g diametric:install
I get a sample config/diametric.yml
to populate with my own connection dataconnect(...some new interface...)
in my models to override the datomic instance in which my model connects to.Nice to have:
Hello,
today I've encountered the weird issue.
I have a model "Badge" in my Datomic db, here's it's ruby's representation:
require 'diametric'
class Badge
include Diametric::Entity
include Diametric::Persistence::Peer
attribute :category, Ref
attribute :date, DateTime
attribute :description, String
attribute :enum_value, String
attribute :enumeration, String, :cardinality => :many
attribute :id, UUID
attribute :name, String
attribute :type, Ref
attribute :value, Double
attribute :week, Integer
attribute :year, Integer
end
Im relying on Peer
API, JRuby.
I want to get the badges with value >= 10.
Following code works fine:
badges = Diametric::Query.new(Badge, nil, true).filter(:>, :value, 9)
the badges.inspect
returns
#<Diametric::Query:0x33de4104 @conditions={}, @filter_values=[9], @conn_or_db=nil, @filters=[[(> ?value ?valuevalue)]], @model=Badge, @resolve=true, @filter_attrs=[:value]>
and I'm free to iterate over set with
badges.each do |badge|
Grape::API.logger.info badge.inspect
end
BUT when I'm trying to use shortcut method
badges = Badge.filter(:>, :value, 9)
the badges.inspect
returns
#<Diametric::Query:0x1e7a60f7 @conditions={}, @filter_values=[9], @conn_or_db=:>, @filters=[[(?value ?valuevalue)]], @model=Badge, @resolve=true, @filter_attrs=[:value]>
(please mind the @conn_or_db
or @filters
keys)
and when i'm trying to iterate over set I'm being given
ArgumentError: The second arg should be a database.
diametric/DiametricPeer.java:270:in `q'
/Users/michalsiemionczyk/.rvm/gems/jruby-1.7.15/gems/diametric-0.1.3-java/lib/diametric/persistence/peer.rb:191:in `q'
/Users/michalsiemionczyk/.rvm/gems/jruby-1.7.15/gems/diametric-0.1.3-java/lib/diametric/query.rb:117:in `each'
/Users/michalsiemionczyk/Projects/yeti-backend/app/services/profiles/filtered_profiles_service.rb:37:in `query_for_date_filtered'
which points to the badges.each do |badge|
.
I guess it might a bug relating to definition of filter
shortcut method,
Modifying the datomic_version.yml file to any version > 0.8.4218 causes the 'rake spec' to fail in a blazing glory of fireworks. Looking at the changelog for 0.8.4254, I don't see anything that stands out as worthy of causing such failure.
Please tell me I'm going something boneheaded, like I need to clear my maven cache... Will it be trivial to bump versions up to the latest?
Can we get more info on this @yokolet?
Use Datomic to its fullest potential. Implement a history API on entities for access an entities historical values.
This feature still needs to be thought out and discussed.
How do you use database functions / transaction functions written in ruby, via diametric?
Currently, it supports very simple functions only. (ex. greater than, less than)
I think we'd better support user defined clojure functions, first, then ruby method.
For example, accepts string form of clojure function that needs one argument. The argument is given as an argument, not in the part of function. Then, creates var type using clojure's read-string function, and so on.
Perhaps, it's better to have arity check.
Can we get more info on this @yokolet?
Is there a way to configure what version of datomic is installed with diametric?
The current repository is configured to a version of datomic that is a few revisions old and we currently need to fork the repository to change it.
https://github.com/relevance/diametric/blob/ae5fa808ae263c95eacb3ad896655e74d6f2a3b2/README.md
Diametric supports both CRuby and JRuby. When Diametric is used on CRuby, Diametric connects to Datomic's REST service. Using Datomic's REST API, Diametric creates schema/data and makes a queries to Datomic. When Diametric is used on JRuby, both Datomic's REST and Peer services are supported. The core parts of Peer service are implemented using Datomic API.
...
The biggest difference between REST and Peer services is available database types. REST service can use memory database only, which means we can't save data in any file as long as using free version. While Peer service can use memory and free version of transactor, which means we can save data in a file through Datomic's transactor.
I found multiple parts of this confusing:
Using Rails 3.2.16, and datomic-free-0.9.4384, my rails application hangs in development mode with webrick when exiting with ctrl-c. If I don't have Datomic running, of course the gem complains with "There is a configuration error with the current diametric.yml. Failed to create connection", but it can quit successfully.
I'm not sure how to begin debugging this issue.
The first step would be built-in tx functions. Using clojure's read-string function, gives it to transaction.
The second step will be to accept user defined tx functions.
The specs that run when the INTEGRATION
env variable is present should run on travis-ci. These specs currently require the datomic rest client to run, so you will need to experiment with writing a ci script that can download, extract and run the datomic rest client (or whatever approach works...).
Currently, a query string holds all attributes for any query. This doesn't matter if a model has a couple of attributes. But, if the model has 100 attributes, 100 attributes will present in the single query string even though only one of attribute is needed.
Another problem is that the query string is generated every time the query happens. This will cause a performance issue especially under the circumstance that millions of queries rush within a minutes.
could you change that to datomic version?
https://github.com/relevance/diametric#datomic-verion
thanks,
M
Hi guys!
I ran into a issue when declaring entities inside modules, which I compiled into a simple gist that demonstrates this problem:
https://gist.github.com/tjsousa/14150b7e798e1e08d316
It fails with the following error:
NoMethodError: undefined method `[]' for nil:NilClass
Anything I am doing wrong here?
Thanks for your time!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.