use logstash as new data ingestion strategy about elasticsearch-river-rethinkdb HOT 18 CLOSED

Hi @frankleng, can you give an example of what you mean?
Do you want to efficiently query a table based on something that incorporates data from a second table?

from elasticsearch-river-rethinkdb.

@danielmewes Probably something like #2 where we could modify the output of what gets synced to ES, rather than just db == index, table == type.

I think it should definitely be possible to sync whatever you want, but I don't know if the river is the right place to do this. Ultimately, once we have the Logstash inputs and outputs written (rethinkdb/rethinkdb#3197), I think that's a better place for this customizability to go

from elasticsearch-river-rethinkdb.

@deontologician sorry didn't see #2 . but yes exactly like that!
#2 + logstash would be godsends to our project. We rely heavily on ES right now to support our search API. but much prefer Rethinkdb as data store.

Hate to keep asking, but any ETA on #2 ?

from elasticsearch-river-rethinkdb.

I think I'm more likely to just implement the Logstash input/outputs and
rely on logstash to handle all of the transformation heavy lifting. Since
rivers are semi-deprecated, that's probably the best route rather than
adding more complicated configuration for the river.

In terms of ETA, I'm hoping early next year to get the Logstash
inputs/outputs. They shouldn't be a huge undertaking to implement.

On Fri, Dec 12, 2014 at 5:53 PM, Frank Leng [email protected]
wrote:

@deontologician https://github.com/deontologician sorry didn't see #2
#2 .
but yes exactly like that!
#2 #2

• logstash would be godsends to our project. We rely heavily on ES right
  now to support our search API. but much prefer Rethinkdb as data store.

Hate to keep asking, but any ETA on #2
#2 ?

—
Reply to this email directly or view it on GitHub
#7 (comment)
.

from elasticsearch-river-rethinkdb.

@deontologician thanks! we are super excited about this.

from elasticsearch-river-rethinkdb.

@deontologician I know you are busy with other things. just wondering if you have any plans in the next month or two to implement this

from elasticsearch-river-rethinkdb.

It's next on my list. I'm currently working on some webui things, but
probably in a week or so.

On Tue, Mar 3, 2015 at 1:02 AM, Frank Leng [email protected] wrote:

@deontologician https://github.com/deontologician I know you are busy
with other things. just wondering if you have any plans in the next month
or two to implement this

—
Reply to this email directly or view it on GitHub
#7 (comment)
.

from elasticsearch-river-rethinkdb.

For us, this is currently a huge hang up. When the river adds the mapping to Elastic, there's not currently a way (to my knowledge) to specify fields that shouldn't be tokenized and analyzed.

This makes it impossible to filter against fields which contain things like e-mail addresses. They'll be tokenized and so matching doesn't work correctly.

from elasticsearch-river-rethinkdb.

I just put in the code review for my previous project, so I'll be picking
this up today
On Mar 12, 2015 21:19, "Kyle Cunningham" [email protected] wrote:

For us, this is currently a huge hang up. When the river adds the mapping
to Elastic, there's not currently a way (to my knowledge) to specify fields
that shouldn't be tokenized and analyzed.

This makes it impossible to filter against fields which contain things
like e-mail addresses. They'll be tokenized and so matching doesn't work
correctly.

—
Reply to this email directly or view it on GitHub
#7 (comment)
.

from elasticsearch-river-rethinkdb.

@codeincarnate I think you can work around it

curl -XPUT '127.0.0.1:9200/table' -d '
{
  "mappings": {
    "table": {
      "properties": {
        "email": {
          "type": "string",
          "index": "not_analyzed"
        }
      }
    }
  }
}

put that mapping in before you initialize the rethinkdb indexes via river.
see http://www.elastic.co/guide/en/elasticsearch/guide/current/mapping-intro.html#custom-field-mappings

from elasticsearch-river-rethinkdb.

@frankleng I tried that, didn't appear to work, though I was extremely tired when doing it. I'll give it another shot.

from elasticsearch-river-rethinkdb.

@codeincarnate cool. be sure to delete existing indices and the _river index before you put in the custom mapping.

from elasticsearch-river-rethinkdb.

Hey guys, I just put the initial (very very beta) version of the logstash plugin up at https://github.com/rethinkdb/logstash-input-rethinkdb/tree/master .

Currently, it needs a logstash filter to be able to pipe directly into ElasticSearch, so this is one piece of the puzzle, not the entire thing. That being said, I'd really appreciate any early feedback you can provide if you're interested in playing around with it.

from elasticsearch-river-rethinkdb.

@deontologician Thanks a lot! I'll definitely test it out this week.
Altho it seems like the two logstash limitations are also present in the river. so no loss there.

from elasticsearch-river-rethinkdb.

@frankleng that's true, they are also present in the river, but I made more of an effort to try reconnecting and to add backfilling behavior since there was no expectation that changefeeds would grow these abilities soon. Since they're on the roadmap now, there is a bit less functionality in the logstash plugin

from elasticsearch-river-rethinkdb.

@deontologician
Tried this and works fine. but as you mentioned it's blocked by the implementation of resumable changefeeds. So we'll stick to the river for now. unless I've missed something.

Thanks!

from elasticsearch-river-rethinkdb.

awesome, thanks for checking it out

On Tue, Jun 2, 2015 at 2:59 PM Frank Leng [email protected] wrote:

@deontologician https://github.com/deontologician
Tried this and works fine. but as you mentioned it's blocked by the
implementation of resumable changefeeds. So we'll stick to the river for
now. unless I've missed something.

Thanks!

—
Reply to this email directly or view it on GitHub
#7 (comment)
.

from elasticsearch-river-rethinkdb.

closing, all discussion about the logstash plugin should continue at https://github.com/rethinkdb/logstash-input-rethinkdb/issues

from elasticsearch-river-rethinkdb.