relevance / diametric Goto Github PK

Currently, Persistence::Peer uses edn-ruby library to convert Ruby object to edn. This is a good idea to keep compatibility over Persistence::REST. However, it loses flexibility to cope with any data type of datomic in Ruby saide.

JRuby can directly create datomic objects using Java API. Eliminating edn-ruby conversion from Persistence::Peer will be good for supporting more data types effectively.

Some are probably implemented, however, needs to make sure. Also, needs to document what queries are correspond to what.

Right now if I do something like:

t = Terrarium.new
t.turtles << Turtle.new(name: "Speedy")
t.save

the turtle is not also saved with the terrarium... I have to create and save the turtle first.

It would be much better if saves were transitive to all new objects, and they were all part of the same datomic transaction.

        irb(main):001:0> require 'diametric'
        NameError: Undefined method == for class 'ActiveSupport::BasicObject'
                      from org/jruby/RubyModule.java:2330:in `undef_method'

Everything's fine with JRuby 1.7.10.

It is easy to create Rails app, but nice to have sample app.

(I have a pullup to fix this; I'm creating the issue so I can tie it to that pull request).

When an association between a parent and a child, all objects save correctly. However if you then load the parent, traverse to the child (causing it to load), the child is marked as dirty. If you then save the parent, a second copy of the child is saved.

Hi,
I'm using Peer Service to connect to the Datomic.
I created Storage Service based on DynamoDB on Amazon by following https://www.youtube.com/watch?v=wG5grJP3jKY tutorial (using CloudFormation service, it's also referenced here http://docs.datomic.com/aws.html).
The tutorial ouptus following URI:
uri = "datomic:ddb://us-east-1/your-system-name/test-db?aws_access_key_id=<MY_ACCESS_KEY>&aws_secret_key=<MY_SECRET_KEY>";
Unfortunately, there's no way for me to put that into URI that connects to the database:

Diametric::Persistence.establish_base_connection({:uri=>"datomic:ddb://us-east-1/your-system-name/test-db?aws_access_key_id=<MY_ACCESS_KEY>&aws_secret_key=<MY_SECRET_KEY>"})

I'm being given
RuntimeError: :db.error/unsupported-protocol Unsupported protocol :ddb

If I change "ddb" into "free" I'm being given
RuntimeError: :db.error/invalid-db-uri Invalid database URI datomic:free://us-east-1/your-system-name/test-db

What would an approach to connect from Diametric directly to remote transactor that's exposed through setting up Datomic by Amazon Cloudformation service? Where do I find that URI?
Regards!

Hello,

I have following models:

    class Badge
      include Diametric::Entity
      include Diametric::Persistence::Peer

      attribute :category, Ref
      attribute :date, DateTime
      attribute :description, String
      attribute :enum_value, String
      attribute :enumeration, String, :cardinality => :many
      attribute :id, UUID
      attribute :name, String
      attribute :type, Ref
      attribute :value, Double
      attribute :week, Integer
      attribute :year, Integer
    end

and

class Person
  include Diametric::Entity
  include Diametric::Persistence::Peer

  attribute :name, String
  attribute :id, UUID, :unique => :identity
  attribute :badges, Ref, :cardinality => :many
  attribute :address, String
  attribute :city, String
  attribute :state, String
  attribute :country, String
  attribute :zip_code, String
  attribute :phone, String
  attribute :email, String
  attribute :age, Integer
  attribute :gender, Ref
end

My intention is to create Person with two Badges.

badge = Badge.new(name: "average_weekly_revenue", value: 30.0, date: DateTime.strptime('2014-10-21T04:05:06', '%Y-%m-%dT%H:%M:%S'))
badge.save

badge2 = Badge.new(name: "some_other_badge", value: 20.0, date: DateTime.strptime('2014-10-21T04:05:06', '%Y-%m-%dT%H:%M:%S'))
badge2.save

so when I create new person:

user_id = UUID.generate
person = Person.new(name: "Natalia Jolie", id: user_id, badges: [badge, badge2])
person.save

I'm being given:

RuntimeError: java.lang.IllegalArgumentException: :db.error/not-an-attribute 17592186045435 is not an attribute.
 get at diametric/DiametricListenableFuture.java:44
save at /Users/michalsiemionczyk/.rvm/gems/jruby-1.7.15/gems/diametric-0.1.3-java/lib/diametric/persistence/peer.rb:15
testD at /Users/michalsiemionczyk/Projects/yeti-backend/michals-test/test.rb:115

And that points to the

 person = Person.new(name: "Natalia Jolie", id: user_id, badges: [badge, badge2])

line. The 17592186045435 number is a dbid of the first badge.

I've tried another approach described in the tutorial in here https://github.com/relevance/diametric#association

user_id = UUID.generate
Person.new(name: "Brad Dipp", id: user_id).save
person = Diametric::Query.new(Person, @conn, true).where(name: "Brad Dipp").first
person.update_attributes(badges: [badge, badge2])
person = Diametric::Query.new(Person, @conn, true).where(name: "Brad Dipp").first

puts person.badges.collect(&:name)

and that throws the same error that points to the person.update_attributes(badges: [badge, badge2]) line.

When I tried the tutorial it throws the same error on me.update_attributes(kids: [mario, luigi]) .

Naturally I'm fine when adding associations with :cardinality => :many, but for my schema I need multiple references.

How to make it work?

When following the wiki, https://github.com/relevance/diametric/wiki/Datomic-Pro-Version-Support, bundle install fails with:

16:37:24 INFO  NaetherTransfer - Downloading: file:/Users/sbuxton/.m2/repository/com/datomic/datomic-pro/0.8.4122/datomic-pro-0.8.4122.pom
16:37:24 INFO  NaetherTransfer - Downloading: http://repo1.maven.org/maven2/com/datomic/datomic-pro/0.8.4122/datomic-pro-0.8.4122.pom
16:37:24 INFO  NaetherTransfer - Downloading: http://clojars.org/repo/com/datomic/datomic-pro/0.8.4122/datomic-pro-0.8.4122.pom
16:37:24 INFO  NaetherTransfer - Downloading: https://repository.jboss.org/nexus/content/groups/public/com/datomic/datomic-pro/0.8.4122/datomic-pro-0.8.4122.pom
16:37:25 INFO  NaetherTransfer - Downloading: http://files.couchbase.com/maven2/com/datomic/datomic-pro/0.8.4122/datomic-pro-0.8.4122.pom
16:37:25 WARN  NaetherRepository - Missing artifact descriptor for com.datomic:datomic-pro:jar:0.8.4122

It appears the datomic pro jar file is no longer hosted in any of the maven repositories.

Currently, REST service is left behind. All tests pass so far, however REST also should have more queries or attribute types and so on.

Hello,
since I'm posting a lot you can expect me to become Diametric master and Diametric well documented library with nice FAQ ;)

I've been struggling with passing more than one argument to Diametric::Persistence::Peer.q method.
Let's say I have a following code:

qString = "
[:find ?e
 :in $ [?badge_name ...]
 :where
 [?e :badge/name ?badge_name]
]

"
It's crucial here that I use collection [?t ...] (I need to create simple OR statetent and retrieve all he Badges that has a name that exist in the set specified by me).

I'm calling it through, eg.

result = Diametric::Persistence::Peer.q(qString, @conn.db, ['facebook_is_fan', 'facebook_engagement_gold', 'average_weekly_revenue'])

Works as expected. However, if I add additional source argument into my query string:

qString = "
[:find ?e
 :in $ ?badge_value [?badge_name ...]
 :where
 [?e :badge/value ?badge_value]
 [?e :badge/name ?badge_name]
]

"
I cannot come up with a way of adding additional parameter as source argument.
For instance,

result = Diametric::Persistence::Peer.q(qString, @conn.db, [[30.42], ['facebook_is_fan', 'facebook_engagement_gold', 'average_weekly_revenue']])

Returns

RuntimeError: Datomic Exception: :db.error/too-few-inputs Query expected 3 inputs but received 2
   q at diametric/DiametricPeer.java:295

Isn't it caused by implementation of q() method that contains the call for Diametric::Persistence::Peer.q(query, db, args.flatten(1)) and the call for Clojure APi should not expect only flattened list of source aguments?

I'd like to achieve sth like the suggested response for this issue https://groups.google.com/forum/?hl=pl#!topic/datomic/QZJrpbp-t-8

Thanks in advanve!

@yokolet and @rkneufeld : you two are doing a great job of working on this gem. If it is ok with you, I am going to change the owner to the Relevance account. Once I do that, could you add me as a collaborator on the new repo?

I would like to filter out for the objects that have , let's say value attribute.
How to perform "lower than A" AND "greater than B" query?
Using query = Diametric::Query.new(Badge, nil, true).filter(:>, :value, 15).filter(:<, :value, 40) won't work, as the query.data.inspect returns

[[:find, #<EDN::Type::Symbol:0x5d27af65 @symbol=:"?e">, :in, #<EDN::Type::Symbol:0x586c3faa @symbol=:"$">, [#<EDN::Type::Symbol:0x2811f8b1 @symbol=:"?valuevalue">, #<EDN::Type::Symbol:0x33de4104 @symbol=:"?valuevalue">], :where, [#<EDN::Type::Symbol:0x4f20b45b @symbol=:"?e">, :"badge/value", #<EDN::Type::Symbol:0x6c20eaf2 @symbol=:"?value">], [(> ?value ?valuevalue)], [(< ?value ?valuevalue)]], [15, 40]]

so in the end both the lower/greater than signs are applied to the same ?valuevalue.
Is there any way I can enforce diametric to take 2 additional arguments (apart from writing a custom query in the string?)

Regards,
Michal

The Getting Started page of the wiki could use some love. Improve this documentation, and when done link README.md to this page.

Refactor Diametric::Persistence such that it becomes the primary module to include and the connection determines different behavior.

A suggestion from @yokolet:

.connect should take an argument :rest or :peer, #peer? and #rest? methods should be on the connection object

For example, given the entity definition and instances below:

class Person
  include Diametric::Entity
  include Diametric::Persistence

  attribute :name, String, :cardinality => :one, :doc => "A person's name"
  attribute :spouse, Ref, :cardinality => :one, :doc => "A person's spouse"
end
Person.create_schema

alice = Person.new(:name => "Alice")
alice.save

bob = Person.new(:name => "Bob", :spouse => alice)
bob.save

alice.spouse= bob
alice.save

currently, there's no easy way to find entity by "a person whose spouse's name is Alice" since spouse method returns dbid.

Currently, diametric Peer API supports a part of datomic API. It should support fully.

Datomic has :db.type/ref type to associate entities. Current Diametric attribute doesn't have the type as well as implementation. So, I'll start working on this.

My idea of refrence type is ...

Definition:

class Person
  include Diametric::Entity

  attribute :name, String, :cardinality => :one, :doc => "A person's name"
  attribute :parent, Ref, :cardinality => :many, :doc => "A person's parent"
end

Person.schema
# Datomic transaction
# [{:db/id #db/id[:db.part/db]
#   :db/ident :person/name
#   :db/valueType :db.type/string
#   :db/cardinality :db.cardinality/one
#   :db/doc "A person's name"
#   :db.install/_attribute :db.part/db}
#
#  {:db/id #db/id[:db.part/db]
#   :db/ident :person/parent
#   :db/valueType :db.type/ref
#   :db/cardinality :db.cardinality/many
#   :db/doc "A person's parent"
#   :db.install/_attribute :db.part/db}

Usage:

alice = Person.new(:name => "Alice")
bob = Person.new(:name => "Bob", :parent => alice)
chris = Person.new(:name => "Chris", :parent => alice)

Does this make sense?

RubyGems.org doesn't report a license for your gem. This is because it is not specified in the gemspec of your last release.

via e.g.

  spec.license = 'MIT'
  # or
  spec.licenses = ['MIT', 'GPL-2']

Including a license in your gemspec is an easy way for rubygems.org and other tools to check how your gem is licensed. As you can imagine, scanning your repository for a LICENSE file or parsing the README, and then attempting to identify the license or licenses is much more difficult and more error prone. So, even for projects that already specify a license, including a license in your gemspec is a good practice. See, for example, how rubygems.org uses the gemspec to display the rails gem license.

There is even a License Finder gem to help companies/individuals ensure all gems they use meet their licensing needs. This tool depends on license information being available in the gemspec. This is an important enough issue that even Bundler now generates gems with a default 'MIT' license.

I hope you'll consider specifying a license in your gemspec. If not, please just close the issue with a nice message. In either case, I'll follow up. Thanks for your time!

Appendix:

If you need help choosing a license (sorry, I haven't checked your readme or looked for a license file), GitHub has created a license picker tool. Code without a license specified defaults to 'All rights reserved'-- denying others all rights to use of the code.
Here's a list of the license names I've found and their frequencies

p.s. In case you're wondering how I found you and why I made this issue, it's because I'm collecting stats on gems (I was originally looking for download data) and decided to collect license metadata,too, and make issues for gemspecs not specifying a license as a public service :). See the previous link or my blog post about this project for more information.

I think this line should be @data.delete_if { |e| e.to_i == entity.dbid }

I can submit a PR if you agree.

And thanks for the gem. It's great for learning datomic in a familiar setting.

Lets say I have an entity, Terrarium, and another entity, Turtle

Terrarium has an attribute defined as:

  attribute :turtles, Ref, cardinality: :many

If I get the terrarium from the database with Terrarium.all, Terrarium.first, or Terrarium.where, all works exactly as expected (thank you!) but if I get the terrarium with find given its id, as in:

t = Terrarium.find 17592186045422

then the terrarium that is returned doesn't have its turtles properly reified... t.turtles will be a Java Set filled with instances of Java::DatomicQuery::EntityMap

I'll spend some time looking at this this weekend, but any hints would be welcome. I was surprised to see find defined in Entity while all the other finder-like things defined in Peer...

I wanted to start a conversation about cardinality/many.

I have a number of thoughts about meshing Ruby/ActiveRecord idioms with Datomics implementation. The most pressing is Datomic's usage of sets (if I'm not mistaken) as opposed to ActiveRecord heavily utilizing ordered and non-unique arrays.

I think it is probably sanest to stay as close to actual datastore implementation as possible. So I'll probably approach this via Set. This may be a little jarring for less Clojure-y developers coming to Datomic, so I think it might be worthwhile to keep the API users informed by complaining early (e.g. you try to assign an array as a default instead of a set).

Next up is retraction/protraction scheme. I think if the set-based approach is used this becomes a simple matter of producing a list of retractions and protractions based upon the difference between original and present values.

REST and Peer raise different exceptions, but both raise an exception when :db.type/uri is specified.

The hard part is to keep track tempids of referencees and referencers.

This implementation can be done by the third argument of tempid function, or use db/unique option to keep track.

In any case, Diametric needs some sort of meta (or helper) class in term of clarity. The save method should save a single entity to datomic.

Hi, sorry if this seems a bit pedantic, but on the project you say you use the MIT license, but then the actual license doesn't look like the MIT license and you have a link to another completely separate license. This is project really released under the MIT license?

Connecting via Peer doesn't seem to support pro storage:

uri = "datomic:ddb://us-east-1/<dynamo-db-name>/<db>?aws_access_key_id=<key>&aws_secret_key=<secret>"
Diametric::Persistence::Peer.create_database(uri)

RuntimeError: Datomic Error: :protocol/unsupported Unsupported protocol :ddb
from diametric/DiametricPeer.java:83:in `create_database'

and

Diametric::Persistence::Peer.connect(uri)

RuntimeError: Failed to create connection
from diametric/DiametricPeer.java:71:in `connect'

Whilst I can connect via the datomic shell using the same uri without issue:

datomic % conn = Peer.connect(uri);
<{:db-id "sampledb-cdda2a0d-3952-49c9-bfb6-adf581d520d7", :unsent-updates-queue 0, :pending-txes 0, :next-t 1000, :basis-t 62, :index-rev 0, :index-root {:rev 0, :key "5293bcf5-617c-438d-a204-6c09d5f6c5a6"}, :log-root {:rev 1, :key "5293bcf9-c604-484c-aed4-2ed33c698074"}}>

Am I missing something here or are there limitations on the diametric Peer library?

A few basic features I want to add:

• Including diametric in a Gemfile will automatically connect on rails boot (if config/diametric.yml is present)
• A generator diametric:install should exist, so when i say: rails g diametric:install I get a sample config/diametric.yml to populate with my own connection data
• I can still declare: connect(...some new interface...) in my models to override the datomic instance in which my model connects to.

Nice to have:

• Override model generators to generate Diametric entities.

Hello,
today I've encountered the weird issue.
I have a model "Badge" in my Datomic db, here's it's ruby's representation:

 require 'diametric'

  class Badge
    include Diametric::Entity
    include Diametric::Persistence::Peer

    attribute :category, Ref
    attribute :date, DateTime
    attribute :description, String
    attribute :enum_value, String
    attribute :enumeration, String, :cardinality => :many
    attribute :id, UUID
    attribute :name, String
    attribute :type, Ref
    attribute :value, Double
    attribute :week, Integer
    attribute :year, Integer
  end

Im relying on Peer API, JRuby.
I want to get the badges with value >= 10.

Following code works fine:
badges = Diametric::Query.new(Badge, nil, true).filter(:>, :value, 9)
the badges.inspect returns
#<Diametric::Query:0x33de4104 @conditions={}, @filter_values=[9], @conn_or_db=nil, @filters=[[(> ?value ?valuevalue)]], @model=Badge, @resolve=true, @filter_attrs=[:value]>

and I'm free to iterate over set with

 badges.each do |badge|
      Grape::API.logger.info badge.inspect
  end

BUT when I'm trying to use shortcut method
badges = Badge.filter(:>, :value, 9)

the badges.inspect returns
#<Diametric::Query:0x1e7a60f7 @conditions={}, @filter_values=[9], @conn_or_db=:>, @filters=[[(?value ?valuevalue)]], @model=Badge, @resolve=true, @filter_attrs=[:value]>

(please mind the @conn_or_db or @filters keys)

and when i'm trying to iterate over set I'm being given

ArgumentError: The second arg should be a database.
    diametric/DiametricPeer.java:270:in `q'
    /Users/michalsiemionczyk/.rvm/gems/jruby-1.7.15/gems/diametric-0.1.3-java/lib/diametric/persistence/peer.rb:191:in `q'
    /Users/michalsiemionczyk/.rvm/gems/jruby-1.7.15/gems/diametric-0.1.3-java/lib/diametric/query.rb:117:in `each'
    /Users/michalsiemionczyk/Projects/yeti-backend/app/services/profiles/filtered_profiles_service.rb:37:in `query_for_date_filtered'

which points to the badges.each do |badge| .

I guess it might a bug relating to definition of filter shortcut method,

Modifying the datomic_version.yml file to any version > 0.8.4218 causes the 'rake spec' to fail in a blazing glory of fireworks. Looking at the changelog for 0.8.4254, I don't see anything that stands out as worthy of causing such failure.

Please tell me I'm going something boneheaded, like I need to clear my maven cache... Will it be trivial to bump versions up to the latest?

Can we get more info on this @yokolet?

Use Datomic to its fullest potential. Implement a history API on entities for access an entities historical values.

This feature still needs to be thought out and discussed.

How do you use database functions / transaction functions written in ruby, via diametric?

Currently, it supports very simple functions only. (ex. greater than, less than)

I think we'd better support user defined clojure functions, first, then ruby method.

For example, accepts string form of clojure function that needs one argument. The argument is given as an argument, not in the part of function. Then, creates var type using clojure's read-string function, and so on.

Perhaps, it's better to have arity check.

Can we get more info on this @yokolet?

Is there a way to configure what version of datomic is installed with diametric?

The current repository is configured to a version of datomic that is a few revisions old and we currently need to fork the repository to change it.

https://github.com/relevance/diametric/blob/ae5fa808ae263c95eacb3ad896655e74d6f2a3b2/README.md

Diametric supports both CRuby and JRuby. When Diametric is used on CRuby, Diametric connects to Datomic's REST service. Using Datomic's REST API, Diametric creates schema/data and makes a queries to Datomic. When Diametric is used on JRuby, both Datomic's REST and Peer services are supported. The core parts of Peer service are implemented using Datomic API.
...
The biggest difference between REST and Peer services is available database types. REST service can use memory database only, which means we can't save data in any file as long as using free version. While Peer service can use memory and free version of transactor, which means we can save data in a file through Datomic's transactor.

I found multiple parts of this confusing:

• "REST service can use memory database only" -- I can't find any documentation about this on the Datomic site. Are more details available?
• "can't save data in any file as long as using free version" -- what if I want to use CRuby with Datomic Pro Starter? Can I persist data to disk in that scenario?

Using Rails 3.2.16, and datomic-free-0.9.4384, my rails application hangs in development mode with webrick when exiting with ctrl-c. If I don't have Datomic running, of course the gem complains with "There is a configuration error with the current diametric.yml. Failed to create connection", but it can quit successfully.

I'm not sure how to begin debugging this issue.

The first step would be built-in tx functions. Using clojure's read-string function, gives it to transaction.

The second step will be to accept user defined tx functions.

The specs that run when the INTEGRATION env variable is present should run on travis-ci. These specs currently require the datomic rest client to run, so you will need to experiment with writing a ci script that can download, extract and run the datomic rest client (or whatever approach works...).

Currently, a query string holds all attributes for any query. This doesn't matter if a model has a couple of attributes. But, if the model has 100 attributes, 100 attributes will present in the single query string even though only one of attribute is needed.

Another problem is that the query string is generated every time the query happens. This will cause a performance issue especially under the circumstance that millions of queries rush within a minutes.

could you change that to datomic version?
https://github.com/relevance/diametric#datomic-verion

thanks,

M

Hi guys!

I ran into a issue when declaring entities inside modules, which I compiled into a simple gist that demonstrates this problem:

https://gist.github.com/tjsousa/14150b7e798e1e08d316

It fails with the following error:

NoMethodError: undefined method `[]' for nil:NilClass

Anything I am doing wrong here?
Thanks for your time!