Allow limiting resultset returned about embulk-input-mongodb HOT 5 CLOSED

Hi.

Currently, this plugin doesn't support limit() as you're saying.
One idea to reduce import data size is using incremental load.

• First execution
  $ embulk run /path/to/config.yml -c diff.yml
• Next execution
  $ embulk run /path/to/config.yml -c diff.yml
  Although you can't reduce import data size at first execution, plugin will get only diff data at next execution.

I think it's possible to implement limit and skip option. However we need to consider about confliction with incremental feature.

e.g.

MongoCursor<Value> cursor = collection
        .find(query)
        .projection(projection)
        .sort(sort)
        .limit(1000)
        .skip(100)

https://github.com/hakobera/embulk-input-mongodb/blob/master/src/main/java/org/embulk/input/mongodb/MongodbInputPlugin.java#L235

from embulk-input-mongodb.

I sent PR to implement limit and skip options. #32

@bcipolli Could you try it?

from embulk-input-mongodb.

@bcipolli @sakama
Sorry for late response. I've released v0.5.1 which includes #32. Please try it, thank you 🙏

from embulk-input-mongodb.

Thanks 👍

from embulk-input-mongodb.

Fixed by #32

from embulk-input-mongodb.