Backfill fails at >1000 docs about elasticsearch-river-rethinkdb HOT 15 CLOSED

Thank you for the report @numso , sorry you ran into this issue.

@deontologician do you have an idea what might be going on here?

from elasticsearch-river-rethinkdb.

It looks like the bulkrequest might be getting failures. Right now the failures are being thrown out. I'd assumed they would raise an exception, but it looks like it returns a status that you need to check and then iterate through the failed items.

@numso I'll create a patched river with that in there, then you can run it and hopefully we'll see what is causing the errors.

from elasticsearch-river-rethinkdb.

Awesome. Thanks for looking into that so quickly!

from elasticsearch-river-rethinkdb.

@numso Could you give me a sample of what your documents look like?

from elasticsearch-river-rethinkdb.

Yup. I was testing with a list of made up Monster's University courses. Here's one:

{
"id": "0284ceb2-b242-4563-9004-b6c1c7eb6a2d" ,
"department": "AV",
"number": "161",
"name": "Flying for Winged Monsters",
"learningUnits": 3
}

from elasticsearch-river-rethinkdb.

@numso I have a build here:
https://dl.dropboxusercontent.com/u/1422975/elasticsearch-river-rethinkdb-1.0.1.jar

I wasn't able to replicate the problem, but hopefully this will expose the issue

from elasticsearch-river-rethinkdb.

Ok thanks. I'll let you know what I find.

from elasticsearch-river-rethinkdb.

I ran the river a couple different times. When trying to backfill 1221 documents, only 999 to 1024 docs make it in (it's different on every run). It feels like an async problem; like maybe the backfill-er shuts down before all the docs actually make it in. My Java's a bit rusty but I'll poke around the code tonight and see if I can help out. It's no fun trying to fix a problem you can't replicate locally ;).

Here are the logs: http://pastebin.com/p3bP2pkH

Here's the test data I'm using http://pastebin.com/1rDpRiFu (111 items total). I just inserted that array into rethink 11 times to make up the 1221 items.

RethinkDB 1.15.2
ElasticSearch 1.3

from elasticsearch-river-rethinkdb.

Very strange!

I made another modification that syncs the backfill request before continuing. Can you try this one out?

https://dl.dropboxusercontent.com/u/1422975/elasticsearch-river-rethinkdb-1.0.1.jar

from elasticsearch-river-rethinkdb.

If you're interested, the branch I built it from is josh/fixes

from elasticsearch-river-rethinkdb.

Hmmm. Tried that new jar and got the same results as before (http://pastebin.com/MkJrNVxw).

I just wanted to double check: I've been replacing the jar in {es_path}/plugins/river-rethinkdb/ with the new ones you've been making, then restarting ES. Is that the right way to reload the plugin or am I missing a step?

from elasticsearch-river-rethinkdb.

You should use the elasticsearch-plugin program to install the jar file. I bumped the version number, so replacing the file and restarting wont work

from elasticsearch-river-rethinkdb.

So I tried:

bin/plugin --install river-rethinkdb --url https://dl.dropboxusercontent.com/u/1422975/elasticsearch-river-rethinkdb-1.0.1.jar

and got this error when running elasticsearch and trying to start a river:

Exception in thread "elasticsearch[Leo][rethinkdb_river][T#1]" java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory

Are you maybe missing some libraries like slf4j in the jar you packaged? Or am I still installing it wrong..

from elasticsearch-river-rethinkdb.

Woo hoo! Your branch worked. I guess I must have been installing your modified plugin incorrectly. I pulled down the branch you're building from, ran mvn clean package, and installed the release that came out. Works like a charm now. Backfilled a rethinkdb table that had 14912 docs and got all 14912.

Thanks so much!

from elasticsearch-river-rethinkdb.

Awesome! Thanks for the big report and trying it out. I'll get an official release out tomorrow

from elasticsearch-river-rethinkdb.