Comments (3)
Just started digging into this. Originally, I was hoping we'd be able to specify an arbitrary ReQL query and pipe the results over to elasticsearch. This might be possible with once rethinkdb/rethinkdb#1118 lands. Until then, I think I've come up with a simpler solution that would fit my use case.
First, we need to support script filters so we can manipulate the documents returned by RethinkDB. This appears to be trivial to implement, see the relavent lines from elasticsearch/elasticsearch-river-couchdb. Second, we need to be able to specify table joins in the plugin configuration. Here's a sample configuration introducing both of these concepts:
{
"type": "rethinkdb",
"rethinkdb": {
"databases": {
"blog": {
"posts": {
"index": "blog",
"type": "posts",
"joins": [
{
"table": "comments",
"field": "comments",
"left_key": "id",
"right_key": "postId"
}
],
"script": "ctx.doc.comments = ctx.doc.comments.values.length"
}
}
}
}
}
In this example we're joining on the table comments
and merging them into the document under the field comments
. Our script then reduces this to just the count. So given this configuration, our indexed documents would look like this:
{
"_id": "261f4990-627b-4844-96ed-08b182121c5e",
"_index": "blog",
"_score": 1.0,
"_source": {
"id": "261f4990-627b-4844-96ed-08b182121c5e",
"body": "cupiditate quo est a modi nesciunt soluta\nipsa voluptas",
"title": "at nam consequatur ea labore ea harum",
"userId": 10.0,
"comments": 5
},
"_type": "posts"
}
from elasticsearch-river-rethinkdb.
Oh, one more thing. It would probably be useful to have a changes:true|false
option on each join indicating whether or not changes to the joined table should trigger a reindex on the parent document. Pinging @deontologician for feedback.
from elasticsearch-river-rethinkdb.
Closing since this won't be implemented in the river, but will be easy to do in logstash using LS filters.
from elasticsearch-river-rethinkdb.
Related Issues (13)
- Incorrect Authorization Key HOT 9
- use logging.yml file to set plugin log level HOT 3
- Mark this deprecated in favor of Logstash plugin
- index does not exists when I do http://localhost:9200/rethinkdbname/rethinkdoc/_search?q=eMail:nikola HOT 3
- elasticsearch-river-rethinkdb has Private Subrepositories
- Elastic Search unable to find plugin when used with Docker HOT 4
- Crash: java.lang.Double cannot be cast to java.lang.String HOT 7
- Backfill fails at >1000 docs HOT 15
- ES 1.4.x support HOT 10
- use logstash as new data ingestion strategy HOT 18
- does this incorporate the rethinkDB changefeed for near-realtime ingestion? HOT 2
- River does not restart after crashing/erroring HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from elasticsearch-river-rethinkdb.