Add script support about elasticsearch-river-rethinkdb HOT 3 CLOSED

Just started digging into this. Originally, I was hoping we'd be able to specify an arbitrary ReQL query and pipe the results over to elasticsearch. This might be possible with once rethinkdb/rethinkdb#1118 lands. Until then, I think I've come up with a simpler solution that would fit my use case.

First, we need to support script filters so we can manipulate the documents returned by RethinkDB. This appears to be trivial to implement, see the relavent lines from elasticsearch/elasticsearch-river-couchdb. Second, we need to be able to specify table joins in the plugin configuration. Here's a sample configuration introducing both of these concepts:

{
  "type": "rethinkdb",
  "rethinkdb": {
    "databases": {
      "blog": {
        "posts": {
          "index": "blog",
          "type": "posts",
          "joins": [
            {
              "table": "comments",
              "field": "comments",
              "left_key": "id",
              "right_key": "postId"
            }
          ],
          "script": "ctx.doc.comments = ctx.doc.comments.values.length"
        }
      }
    }
  }
}

In this example we're joining on the table comments and merging them into the document under the field comments. Our script then reduces this to just the count. So given this configuration, our indexed documents would look like this:

{
  "_id": "261f4990-627b-4844-96ed-08b182121c5e",
  "_index": "blog",
  "_score": 1.0,
  "_source": {
    "id": "261f4990-627b-4844-96ed-08b182121c5e",
    "body": "cupiditate quo est a modi nesciunt soluta\nipsa voluptas",
    "title": "at nam consequatur ea labore ea harum",
    "userId": 10.0,
    "comments": 5
  },
  "_type": "posts"
}

from elasticsearch-river-rethinkdb.

Oh, one more thing. It would probably be useful to have a changes:true|false option on each join indicating whether or not changes to the joined table should trigger a reindex on the parent document. Pinging @deontologician for feedback.

from elasticsearch-river-rethinkdb.

Closing since this won't be implemented in the river, but will be easy to do in logstash using LS filters.

from elasticsearch-river-rethinkdb.