what does throttlesize do? about elasticsearch-river-mongodb HOT 6 CLOSED

throttlesize defines the size of the queue where mongoDB documents are stored in memory before to be ingested by Elasticsearch. The default value is 500.
The value could be increased if a high volume of MongoDB documents has to be imported.

from elasticsearch-river-mongodb.

Thanks. What happens when the queue's full?

from elasticsearch-river-mongodb.

It waits until a space is available in the blocking queue.

Thanks,
Richard.

On Thu, Jan 17, 2013 at 7:26 AM, j0hnsmith [email protected] wrote:

Thanks. What happens when the queue's full?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/55#issuecomment-12365930.

from elasticsearch-river-mongodb.

I'm not quite sure I understand you, what waits?

When objects are being updated in mongodb faster than they can be processed by elasticsearch and the queue is full, will those updates that won't fit in the queue eventually get into elasticsearch (once there is room in the queue)? How/where are the updates that don't fit in the queue stored?

For clarity, are there any circumstances (apart from crashes etc) where updates to objects in mongodb don't make it into elasticsearch due to load issues (eg are dropped)?

Thanks for your help

from elasticsearch-river-mongodb.

If the queue is full its thread will be blocked until the operation (put a new item) can succeed.
Meanwhile a different thread process the queue.

There should be no data lost even if crashes happened.
The logic is based on last update time (extracted from oplog.rs) this value is updated in elasticsearch river metadata only when the associated document has been successfully inserted.

from elasticsearch-river-mongodb.

Thanks

from elasticsearch-river-mongodb.