No known previous slurping time about elasticsearch-river-mongodb HOT 23 CLOSED

Hi,

What version of ElasticSearch and MongoDB are you using?
Describe your MongoDB configuration (cluster, authentication, switch in the command line).

Please send elasticsearch.log

Thanks,
Richard.

------Original Message------
From: Peter Ivie
To: Richard Louapre
Subject: [elasticsearch-river-mongodb] No known previous slurping time (#15)
Sent: May 29, 2012 11:11 AM

I tried to follow these instructions exactly:
https://gist.github.com/2029361

But when I run:
curl -XGET "http://localhost:9200/testmongo/_search?q=firstName:John"
I get:
{"error":"IndexMissingException[[testmongo] missing]","status":404}

The elasticsearch log just keeps repeating this:

java.util.NoSuchElementException
at java.util.LinkedList$ListItr.next(LinkedList.java:698)
at com.mongodb.DBCursor._next(DBCursor.java:453)
at com.mongodb.DBCursor.next(DBCursor.java:533)
at org.elasticsearch.river.mongodb.MongoDBRiver$Slurper.processFullCollection(MongoDBRiver.java:378)
at org.elasticsearch.river.mongodb.MongoDBRiver$Slurper.run(MongoDBRiver.java:353)
at java.lang.Thread.run(Thread.java:680)
[2012-05-29 02:29:43,249][INFO ][river.mongodb ] [Node1] [mongodb][mongodb] No known previous slurping time for this collection
[2012-05-29 02:29:43,252][INFO ][node ] [Node1] {0.19.3}[5532]: stopping ...
[2012-05-29 02:29:43,261][INFO ][river.mongodb ] [Node1] [mongodb][mongodb] closing mongodb stream river
[2012-05-29 02:29:43,270][WARN ][river.mongodb ] [Node1] [mongodb][mongodb] A mongoDB cursor bug ?

And the mongodb log just keeps repeating this:
Tue May 29 02:29:43 [conn3] CMD fsync: sync:1 lock:1
Tue May 29 02:29:43 [conn3] removeJournalFiles
Tue May 29 02:29:43 [fsyncjob] db is now locked for snapshotting, no writes allowed. db.fsyncUnlock() to unlock
Tue May 29 02:29:43 [fsyncjob] For more info see http://www.mongodb.org/display/DOCS/fsync+Command
Tue May 29 02:29:43 [conn3] command: unlock requested

Any ideas on what I am doing wrong?

Reply to this email directly or view it on GitHub:
#15

Sent via BlackBerry Bold

from elasticsearch-river-mongodb.

Elasticsearch 0.19.3
MongoDB 2.0.5
Both installed with 'brew install' on my mac and using default
configuration (except for node name). Started/stopped using 'lunchy'.

plugin -install elasticsearch/elasticsearch-mapper-attachments/1.2.0
(have also tried the default version)

plugin -install richardwilly98/elasticsearch-river-mongodb/1.1.0 (have
also tried the default version)

lunchy start elasticsearch

Elasticsearch log so far:
[2012-05-29 16:31:01,616][INFO ][node ] [devbox]
{0.19.3}[6620]: initializing ...
[2012-05-29 16:31:01,640][INFO ][plugins ] [devbox]
loaded [river-mongodb, mapper-attachments], sites []
[2012-05-29 16:31:03,483][INFO ][node ] [devbox]
{0.19.3}[6620]: initialized
[2012-05-29 16:31:03,483][INFO ][node ] [devbox]
{0.19.3}[6620]: starting ...
[2012-05-29 16:31:03,593][INFO ][transport ] [devbox]
bound_address {inet[/0.0.0.0:9300]}, publish_address
{inet[/192.168.2.4:9300]}
[2012-05-29 16:31:06,700][INFO ][cluster.service ] [devbox]
new_master [devbox][nX3rgn0OSzm1dmPuzVAJKQ][inet[/192.168.2.4:9300]],
reason: zen-disco-join (elected_as_master)
[2012-05-29 16:31:06,754][INFO ][discovery ] [devbox]
elasticsearch_peterivie/nX3rgn0OSzm1dmPuzVAJKQ
[2012-05-29 16:31:06,770][INFO ][http ] [devbox]
bound_address {inet[/0.0.0.0:9200]}, publish_address
{inet[/192.168.2.4:9200]}
[2012-05-29 16:31:06,770][INFO ][node ] [devbox]
{0.19.3}[6620]: started
[2012-05-29 16:31:06,772][INFO ][indices.store ] [devbox]
[mongoindex] dangling index, exists on local file system, but not in
cluster metadata, scheduling to delete in [2h]
[2012-05-29 16:31:06,773][INFO ][gateway ] [devbox]
recovered [0] indices into cluster_state

I made sure that testmongo.person exists on MongoDB but the collection
is empty (I get lockups otherwise).

#!/bin/bash
curl -X PUT 'localhost:9200/_river/mongodb/_meta' -d '{
"type": "mongodb",
"mongodb": {
"db": "testmongo",
"collection": "person",
"host": "127.0.0.1",
"port": "27017"
},
"index": {
"name": "mongoindex",
"type": "person"
}
}'

The log goes crazy at this point with continual repetition (every 20
ms) of the following:
[2012-05-29 16:41:00,272][INFO ][river.mongodb ] [devbox]
[mongodb][mongodb] No known previous slurping time for this collection
[2012-05-29 16:41:00,293][WARN ][river.mongodb ] [devbox]
[mongodb][mongodb] A mongoDB cursor bug ?
java.util.NoSuchElementException
at java.util.LinkedList$ListItr.next(LinkedList.java:698)
at com.mongodb.DBCursor._next(DBCursor.java:453)
at com.mongodb.DBCursor.next(DBCursor.java:533)
at org.elasticsearch.river.mongodb.MongoDBRiver$Slurper.processFullCollection(MongoDBRiver.java:378)
at org.elasticsearch.river.mongodb.MongoDBRiver$Slurper.run(MongoDBRiver.java:353)
at java.lang.Thread.run(Thread.java:680)

devbox:plugins peterivie$ mongo
MongoDB shell version: 2.0.5
connecting to: test

use testmongo
switched to db testmongo
var p = {firstName: "John", lastName: "Doe"}
db.person.save(p)
exit
bye

And the elasticsearch log continues to spin.

On Tue, May 29, 2012 at 3:47 PM, Richard Louapre
[email protected]
wrote:

Hi,

What version of ElasticSearch and MongoDB are you using?
Describe your MongoDB configuration (cluster, authentication, switch in the command line).

Please send elasticsearch.log

Thanks,
  Richard.

------Original Message------
From: Peter Ivie
To: Richard Louapre
Subject: [elasticsearch-river-mongodb] No known previous slurping time (#15)
Sent: May 29, 2012 11:11 AM

I tried to follow these instructions exactly:
https://gist.github.com/2029361

But when I run:
curl -XGET "http://localhost:9200/testmongo/_search?q=firstName:John"
I get:
{"error":"IndexMissingException[[testmongo] missing]","status":404}

The elasticsearch log just keeps repeating this:

java.util.NoSuchElementException
       at java.util.LinkedList$ListItr.next(LinkedList.java:698)
       at com.mongodb.DBCursor._next(DBCursor.java:453)
       at com.mongodb.DBCursor.next(DBCursor.java:533)
       at org.elasticsearch.river.mongodb.MongoDBRiver$Slurper.processFullCollection(MongoDBRiver.java:378)
       at org.elasticsearch.river.mongodb.MongoDBRiver$Slurper.run(MongoDBRiver.java:353)
       at java.lang.Thread.run(Thread.java:680)
[2012-05-29 02:29:43,249][INFO ][river.mongodb            ] [Node1] [mongodb][mongodb] No known previous slurping time for this collection
[2012-05-29 02:29:43,252][INFO ][node                     ] [Node1] {0.19.3}[5532]: stopping ...
[2012-05-29 02:29:43,261][INFO ][river.mongodb            ] [Node1] [mongodb][mongodb] closing mongodb stream river
[2012-05-29 02:29:43,270][WARN ][river.mongodb            ] [Node1] [mongodb][mongodb] A mongoDB cursor bug ?

And the mongodb log just keeps repeating this:
Tue May 29 02:29:43 [conn3] CMD fsync:  sync:1 lock:1
Tue May 29 02:29:43 [conn3] removeJournalFiles
Tue May 29 02:29:43 [fsyncjob] db is now locked for snapshotting, no writes allowed. db.fsyncUnlock() to unlock
Tue May 29 02:29:43 [fsyncjob]     For more info see http://www.mongodb.org/display/DOCS/fsync+Command
Tue May 29 02:29:43 [conn3] command: unlock requested

Any ideas on what I am doing wrong?

---

Reply to this email directly or view it on GitHub:
#15

Sent via BlackBerry Bold

---

Reply to this email directly or view it on GitHub:
#15 (comment)

from elasticsearch-river-mongodb.

I tried
plugin -install richardwilly98/elasticsearch-river-mongodb
so that I could change the java code, but then it can't find the mongo
java driver, and I can't figure out where to install it.

On Tue, May 29, 2012 at 4:44 PM, Peter Ivie [email protected] wrote:

Elasticsearch 0.19.3
MongoDB 2.0.5
Both installed with 'brew install' on my mac and using default
configuration (except for node name). Started/stopped using 'lunchy'.

plugin -install elasticsearch/elasticsearch-mapper-attachments/1.2.0
(have also tried the default version)

plugin -install richardwilly98/elasticsearch-river-mongodb/1.1.0 (have
also tried the default version)

lunchy start elasticsearch

Elasticsearch log so far:
[2012-05-29 16:31:01,616][INFO ][node                     ] [devbox]
{0.19.3}[6620]: initializing ...
[2012-05-29 16:31:01,640][INFO ][plugins                  ] [devbox]
loaded [river-mongodb, mapper-attachments], sites []
[2012-05-29 16:31:03,483][INFO ][node                     ] [devbox]
{0.19.3}[6620]: initialized
[2012-05-29 16:31:03,483][INFO ][node                     ] [devbox]
{0.19.3}[6620]: starting ...
[2012-05-29 16:31:03,593][INFO ][transport                ] [devbox]
bound_address {inet[/0.0.0.0:9300]}, publish_address
{inet[/192.168.2.4:9300]}
[2012-05-29 16:31:06,700][INFO ][cluster.service          ] [devbox]
new_master [devbox][nX3rgn0OSzm1dmPuzVAJKQ][inet[/192.168.2.4:9300]],
reason: zen-disco-join (elected_as_master)
[2012-05-29 16:31:06,754][INFO ][discovery                ] [devbox]
elasticsearch_peterivie/nX3rgn0OSzm1dmPuzVAJKQ
[2012-05-29 16:31:06,770][INFO ][http                     ] [devbox]
bound_address {inet[/0.0.0.0:9200]}, publish_address
{inet[/192.168.2.4:9200]}
[2012-05-29 16:31:06,770][INFO ][node                     ] [devbox]
{0.19.3}[6620]: started
[2012-05-29 16:31:06,772][INFO ][indices.store            ] [devbox]
[mongoindex] dangling index, exists on local file system, but not in
cluster metadata, scheduling to delete in [2h]
[2012-05-29 16:31:06,773][INFO ][gateway                  ] [devbox]
recovered [0] indices into cluster_state

I made sure that testmongo.person exists on MongoDB but the collection
is empty (I get lockups otherwise).

#!/bin/bash
curl -X PUT 'localhost:9200/_river/mongodb/_meta' -d '{
 "type": "mongodb",
 "mongodb": {
   "db": "testmongo",
   "collection": "person",
   "host": "127.0.0.1",
   "port": "27017"
 },
 "index": {
   "name": "mongoindex",
   "type": "person"
 }
}'

The log goes crazy at this point with continual repetition (every 20
ms) of the following:
[2012-05-29 16:41:00,272][INFO ][river.mongodb            ] [devbox]
[mongodb][mongodb] No known previous slurping time for this collection
[2012-05-29 16:41:00,293][WARN ][river.mongodb            ] [devbox]
[mongodb][mongodb] A mongoDB cursor bug ?
java.util.NoSuchElementException
       at java.util.LinkedList$ListItr.next(LinkedList.java:698)
       at com.mongodb.DBCursor._next(DBCursor.java:453)
       at com.mongodb.DBCursor.next(DBCursor.java:533)
       at org.elasticsearch.river.mongodb.MongoDBRiver$Slurper.processFullCollection(MongoDBRiver.java:378)
       at org.elasticsearch.river.mongodb.MongoDBRiver$Slurper.run(MongoDBRiver.java:353)
       at java.lang.Thread.run(Thread.java:680)

devbox:plugins peterivie$ mongo
MongoDB shell version: 2.0.5
connecting to: test

use testmongo
switched to db testmongo
var p = {firstName: "John", lastName: "Doe"}
db.person.save(p)
exit
bye

And the elasticsearch log continues to spin.

On Tue, May 29, 2012 at 3:47 PM, Richard Louapre
[email protected]
wrote:

Hi,

What version of ElasticSearch and MongoDB are you using?
Describe your MongoDB configuration (cluster, authentication, switch in the command line).

Please send elasticsearch.log

Thanks,
  Richard.

------Original Message------
From: Peter Ivie
To: Richard Louapre
Subject: [elasticsearch-river-mongodb] No known previous slurping time (#15)
Sent: May 29, 2012 11:11 AM

I tried to follow these instructions exactly:
https://gist.github.com/2029361

But when I run:
curl -XGET "http://localhost:9200/testmongo/_search?q=firstName:John"
I get:
{"error":"IndexMissingException[[testmongo] missing]","status":404}

The elasticsearch log just keeps repeating this:

java.util.NoSuchElementException
       at java.util.LinkedList$ListItr.next(LinkedList.java:698)
       at com.mongodb.DBCursor._next(DBCursor.java:453)
       at com.mongodb.DBCursor.next(DBCursor.java:533)
       at org.elasticsearch.river.mongodb.MongoDBRiver$Slurper.processFullCollection(MongoDBRiver.java:378)
       at org.elasticsearch.river.mongodb.MongoDBRiver$Slurper.run(MongoDBRiver.java:353)
       at java.lang.Thread.run(Thread.java:680)
[2012-05-29 02:29:43,249][INFO ][river.mongodb            ] [Node1] [mongodb][mongodb] No known previous slurping time for this collection
[2012-05-29 02:29:43,252][INFO ][node                     ] [Node1] {0.19.3}[5532]: stopping ...
[2012-05-29 02:29:43,261][INFO ][river.mongodb            ] [Node1] [mongodb][mongodb] closing mongodb stream river
[2012-05-29 02:29:43,270][WARN ][river.mongodb            ] [Node1] [mongodb][mongodb] A mongoDB cursor bug ?

And the mongodb log just keeps repeating this:
Tue May 29 02:29:43 [conn3] CMD fsync:  sync:1 lock:1
Tue May 29 02:29:43 [conn3] removeJournalFiles
Tue May 29 02:29:43 [fsyncjob] db is now locked for snapshotting, no writes allowed. db.fsyncUnlock() to unlock
Tue May 29 02:29:43 [fsyncjob]     For more info see http://www.mongodb.org/display/DOCS/fsync+Command
Tue May 29 02:29:43 [conn3] command: unlock requested

Any ideas on what I am doing wrong?

---

Reply to this email directly or view it on GitHub:
#15

Sent via BlackBerry Bold

---

Reply to this email directly or view it on GitHub:
#15 (comment)

from elasticsearch-river-mongodb.

Hi,

Could please try that.

Stop ES
Delete %ES_HOME%\plugins\river-mongodb folder
Execute %ES_HOME%\bin\plugin.bat -install richardwilly98/elasticsearch -river-mongodb/1.3.0
Update %ES_HOME%\config\logging.yml Add river.mongodb: DEBUG in logger: section
Start ES
Try again.

Please send me elasticsearch.log if not successful.

from elasticsearch-river-mongodb.

I had to re-install elasticsearch and it installed 0.19.4. I did everything with the newer version of river-mongodb and the debug line. No spinning now so that is good, but I still get an error when querying the index:

$ curl -XGET 'http://localhost:9200/testmongo/_search?q=firstName:John'
{"error":"IndexMissingException[[testmongo] missing]","status":404}

Here is the log file:
[2012-06-06 05:03:55,781][INFO ][node ] [Sea Urchin] {0.19.4}[40401]: initializing ...
[2012-06-06 05:03:55,803][INFO ][plugins ] [Sea Urchin] loaded [river-mongodb, mapper-attachments], sites []
[2012-06-06 05:03:57,615][INFO ][node ] [Sea Urchin] {0.19.4}[40401]: initialized
[2012-06-06 05:03:57,615][INFO ][node ] [Sea Urchin] {0.19.4}[40401]: starting ...
[2012-06-06 05:03:57,726][INFO ][transport ] [Sea Urchin] bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/192.168.2.4:9300]}
[2012-06-06 05:04:00,835][INFO ][cluster.service ] [Sea Urchin] new_master [Sea Urchin][RKvjkQZVQu2tzXb-MtKYmg][inet[/192.168.2.4:9300]], reason: zen-disco-join (elected_as_master)
[2012-06-06 05:04:00,888][INFO ][discovery ] [Sea Urchin] elasticsearch_peterivie/RKvjkQZVQu2tzXb-MtKYmg
[2012-06-06 05:04:00,905][INFO ][http ] [Sea Urchin] bound_address {inet[/0.0.0.0:9200]}, publish_address {inet[/192.168.2.4:9200]}
[2012-06-06 05:04:00,905][INFO ][node ] [Sea Urchin] {0.19.4}[40401]: started
[2012-06-06 05:04:00,906][INFO ][gateway ] [Sea Urchin] recovered [0] indices into cluster_state
[2012-06-06 05:05:52,030][INFO ][cluster.metadata ] [Sea Urchin] [_river] creating index, cause [auto(index api)], shards [1]/[1], mappings []
[2012-06-06 05:05:52,547][INFO ][cluster.metadata ] [Sea Urchin] [_river] update_mapping mongodb
[2012-06-06 05:05:52,573][DEBUG][river.mongodb ] [Sea Urchin] [mongodb][mongodb] Prefix: [Sea Urchin] [mongodb][mongodb] - name: river.mongodb
[2012-06-06 05:05:52,574][INFO ][river.mongodb ] [Sea Urchin] [mongodb][mongodb] starting mongodb stream: host [127.0.0.1], port [27017], gridfs [false], filter [testmongo], db [mongoindex], indexing to [person]/[{}]
[2012-06-06 05:05:52,616][INFO ][cluster.metadata ] [Sea Urchin] [mongoindex] creating index, cause [api], shards [5]/[1], mappings []
[2012-06-06 05:05:52,876][INFO ][cluster.metadata ] [Sea Urchin] [_river] update_mapping mongodb
[2012-06-06 05:05:52,901][ERROR][river.mongodb ] [Sea Urchin] [mongodb][mongodb] Cannot find oplog.rs collection. Please use check this link: http://goo.gl/2x5IW

from elasticsearch-river-mongodb.

Hi,

See the last line in the log. The replicaset has not been enabled in MongoDB.

Please check the link: http://goo.gl/2x5IW

The minimum configuration in MongoDB would be --replSet switch and enable the replicaset using rs.initialize()

Thanks,
Richard.
Sent via BlackBerry Bold

-----Original Message-----
From: Peter Ivie [email protected]
Date: Wed, 6 Jun 2012 03:15:34
To: Richard [email protected]
Subject: Re: [elasticsearch-river-mongodb] No known previous slurping time (#15)

I had to re-install elasticsearch and it installed 0.19.4. I did everything with the newer version of river-mongodb and the debug line. No spinning now so that is good, but I still get an error when querying the index:

$ curl -XGET 'http://localhost:9200/testmongo/_search?q=firstName:John'
{"error":"IndexMissingException[[testmongo] missing]","status":404}

Here is the log file:
[2012-06-06 05:03:55,781][INFO ][node ] [Sea Urchin] {0.19.4}[40401]: initializing ...
[2012-06-06 05:03:55,803][INFO ][plugins ] [Sea Urchin] loaded [river-mongodb, mapper-attachments], sites []
[2012-06-06 05:03:57,615][INFO ][node ] [Sea Urchin] {0.19.4}[40401]: initialized
[2012-06-06 05:03:57,615][INFO ][node ] [Sea Urchin] {0.19.4}[40401]: starting ...
[2012-06-06 05:03:57,726][INFO ][transport ] [Sea Urchin] bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/192.168.2.4:9300]}
[2012-06-06 05:04:00,835][INFO ][cluster.service ] [Sea Urchin] new_master [Sea Urchin][RKvjkQZVQu2tzXb-MtKYmg][inet[/192.168.2.4:9300]], reason: zen-disco-join (elected_as_master)
[2012-06-06 05:04:00,888][INFO ][discovery ] [Sea Urchin] elasticsearch_peterivie/RKvjkQZVQu2tzXb-MtKYmg
[2012-06-06 05:04:00,905][INFO ][http ] [Sea Urchin] bound_address {inet[/0.0.0.0:9200]}, publish_address {inet[/192.168.2.4:9200]}
[2012-06-06 05:04:00,905][INFO ][node ] [Sea Urchin] {0.19.4}[40401]: started
[2012-06-06 05:04:00,906][INFO ][gateway ] [Sea Urchin] recovered [0] indices into cluster_state
[2012-06-06 05:05:52,030][INFO ][cluster.metadata ] [Sea Urchin] [_river] creating index, cause [auto(index api)], shards [1]/[1], mappings []
[2012-06-06 05:05:52,547][INFO ][cluster.metadata ] [Sea Urchin] [_river] update_mapping mongodb
[2012-06-06 05:05:52,573][DEBUG][river.mongodb ] [Sea Urchin] [mongodb][mongodb] Prefix: [Sea Urchin] [mongodb][mongodb] - name: river.mongodb
[2012-06-06 05:05:52,574][INFO ][river.mongodb ] [Sea Urchin] [mongodb][mongodb] starting mongodb stream: host [127.0.0.1], port [27017], gridfs [false], filter [testmongo], db [mongoindex], indexing to [person]/[{}]
[2012-06-06 05:05:52,616][INFO ][cluster.metadata ] [Sea Urchin] [mongoindex] creating index, cause [api], shards [5]/[1], mappings []
[2012-06-06 05:05:52,876][INFO ][cluster.metadata ] [Sea Urchin] [_river] update_mapping mongodb
[2012-06-06 05:05:52,901][ERROR][river.mongodb ] [Sea Urchin] [mongodb][mongodb] Cannot find oplog.rs collection. Please use check this link: http://goo.gl/2x5IW

Reply to this email directly or view it on GitHub:
#15 (comment)

from elasticsearch-river-mongodb.

Hello,

I believe that I am experiencing the same issue.

ElasticSearch: 0.19.7
JVM: 22.0-b10
Mongo-River: 1.4.0
Mongo-Driver: 2.8.0

The Mongo collection has around 2M records.
The Elastic index currently has around 10M records.

I create my river with:

curl -X PUT 'http://testnode01:9200/_river/test_data_type1/_meta' -d '{
"type": "mongodb",
"mongodb": {
"host":"mongodb01",
"db": "test_data",
"collection": "type1",
"gridfs": true
},
"index": {
"name": "test_data",
"type": "test_data_type1"
}
}'

I am seeing this in the logs:

[2012-08-03 09:40:03,596][INFO ][river.mongodb] [Glamor] [mongodb][test_data_type1] No known previous slurping time for this collection
[2012-08-03 09:40:04,802][ERROR][river.mongodb [Glamor] [mongodb][test_data_type1] Mongo gave an exception
com.mongodb.MongoException: capped cursor overrun during query: local.oplog.rs
at com.mongodb.MongoException.parse(MongoException.java:82)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:317)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:298)
at com.mongodb.DBCursor._check(DBCursor.java:369)
at com.mongodb.DBCursor._next(DBCursor.java:451)
at com.mongodb.DBCursor.next(DBCursor.java:527)
at org.elasticsearch.river.mongodb.MongoDBRiver$Slurper.processFullCollection(MongoDBRiver.java:580)
at org.elasticsearch.river.mongodb.MongoDBRiver$Slurper.run(MongoDBRiver.java:553)
at java.lang.Thread.run(Thread.java:722)
[2012-08-03 09:40:04,828][INFO ][river.mongodb] [Glamor] [mongodb][test_data_type1] No known previous slurping time for this collection

The pattern of exception happends a few times per second and then stops after a minute or two. I guess it gives up.

Does anyone have a suggestion ?

from elasticsearch-river-mongodb.

Hi,

Please test with version 1.5.0. I was able to index 12M documents on my laptop with no issue.
Feedback is welcome.

Thanks,
Richard.

from elasticsearch-river-mongodb.

Hi Richard,

I came across this issue while I'm integrating river 1.5.0 with ES v11. My mongodb version is 2.2.0, is there any possibility that my mongodb made this happening? Or the river changes? BTW, i'm using mapper-attachments 1.6.

Thanks,
Spancer

from elasticsearch-river-mongodb.

Hi Spancer,

Can you please provide ES log file?
How many documents in the collection?
What river settings did you use?

Thanks,
Richard.

On Thu, Nov 8, 2012 at 10:27 PM, spancer [email protected] wrote:

Hi Richard,

I came across this issue while I'm integrating river 1.5.0 with ES v11. My
mongodb version is 2.2.0, is there any possibility that my mongodb made
this happening? Or the river changes? BTW, i'm using mapper-attachments
1.6.

Thanks,
Spancer

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/15#issuecomment-10214504.

from elasticsearch-river-mongodb.

Hi Richard,

Thanks for the concern on this topic.

For this issue, I used the latest file (https://github.com/downloads/richardwilly98/elasticsearch-river-mongodb/elasticsearch-river-mongodb-1.6.0-SNAPSHOT-plugin.zip), but after I built up the river, it just hang there, no data indiced to ES. And I got IndexShardMissingException.

Log info (using elasticsearch-river-mongodb-1.6.0-SNAPSHOT.jar) :

[2012-11-14 11:22:23,125][INFO ][ansj-analyzer ] [Dict Loading] F:\java\es\elasticsearch-0.19.11\config\ansj\userLibrary.dic,UserDict Time:922
[2012-11-14 11:22:23,953][INFO ][ansj-analyzer ] [Dict Loading] F:\java\es\elasticsearch-0.19.11\config\ansj\userLibrary.dic,UserDict Time:828
[2012-11-14 11:22:23,968][INFO ][cluster.metadata ] [226-ES-11] [pagerscn] creating index, cause [api], shards [3]/[0], mappings [pagerscn]
[2012-11-14 11:22:24,109][DEBUG][action.admin.indices.status] [226-ES-11] [pagerscn][1], node[eMigkQC9QBKwmw15y73bPA], [P], s[INITIALIZING]: Failed to execute [org.elasticsearch.action.admin.indices.status.IndicesStatusRequest@133badd]
org.elasticsearch.index.IndexShardMissingException: [pagerscn][1] missing
at org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at org.elasticsearch.action.admin.indices.status.TransportIndicesStatusAction.shardOperation(TransportIndicesStatusAction.java:153)
at org.elasticsearch.action.admin.indices.status.TransportIndicesStatusAction.shardOperation(TransportIndicesStatusAction.java:59)
at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234)
at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211)
at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-11-14 11:22:24,109][DEBUG][action.admin.indices.status] [226-ES-11] [pagerscn][2], node[eMigkQC9QBKwmw15y73bPA], [P], s[INITIALIZING]: Failed to execute [org.elasticsearch.action.admin.indices.status.IndicesStatusRequest@133badd]
org.elasticsearch.index.IndexShardMissingException: [pagerscn][2] missing
at org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at org.elasticsearch.action.admin.indices.status.TransportIndicesStatusAction.shardOperation(TransportIndicesStatusAction.java:153)
at org.elasticsearch.action.admin.indices.status.TransportIndicesStatusAction.shardOperation(TransportIndicesStatusAction.java:59)
at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234)
at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211)
at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-11-14 11:22:24,468][INFO ][river.mongodb ] [226-ES-11] [mongodb][mpagerscn] No known previous slurping time for this collection
[2012-11-14 11:22:24,468][DEBUG][river.mongodb ] [226-ES-11] [mongodb][mpagerscn] Using filter: { "$and" : [ { "ns" : "proddb.wddata_cn_qklw"}]}
[2012-11-14 11:22:24,578][INFO ][cluster.metadata ] [226-ES-11] [_river] update_mapping mpagerscn

However, when I stopped ES service, changed the 1.6.0 river jar to elasticsearch-river-mongodb-1.4.0-SNAPSHOT.jar, and restart the ES service, it works. Data polled in and indeced successfully.

The new log info :

[2012-11-14 11:29:07,562][WARN ][bootstrap ] jvm uses the client vm, make sure to run java with the server vm for best performance by adding -server to the command line
[2012-11-14 11:29:07,640][INFO ][node ] [226-ES-11] {0.19.11}[20200]: initializing ...
[2012-11-14 11:29:07,671][INFO ][plugins ] [226-ES-11] loaded [river-mongodb, mapper-attachments, analysis-ansj, jdbc-river], sites []
[2012-11-14 11:29:09,234][INFO ][node ] [226-ES-11] {0.19.11}[20200]: initialized
[2012-11-14 11:29:09,234][INFO ][node ] [226-ES-11] {0.19.11}[20200]: starting ...
[2012-11-14 11:29:09,312][INFO ][transport ] [226-ES-11] bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/192.168.1.226:9300]}
[2012-11-14 11:29:14,359][INFO ][cluster.service ] [226-ES-11] new_master [226-ES-11][cZVOSDaPQyCZVxiTtA824A][inet[/192.168.1.226:9300]]{master=true}, reason: zen-disco-join (elected_as_master)
[2012-11-14 11:29:14,375][INFO ][discovery ] [226-ES-11] testcluster/cZVOSDaPQyCZVxiTtA824A
[2012-11-14 11:29:14,390][INFO ][http ] [226-ES-11] bound_address {inet[/0.0.0.0:9200]}, publish_address {inet[/192.168.1.226:9200]}
[2012-11-14 11:29:14,390][INFO ][node ] [226-ES-11] {0.19.11}[20200]: started
[2012-11-14 11:29:15,328][INFO ][ansj-analyzer ] [Dict Loading] F:\java\es\elasticsearch-0.19.11\config\ansj\userLibrary.dic,UserDict Time:860
[2012-11-14 11:29:16,093][INFO ][ansj-analyzer ] [Dict Loading] F:\java\es\elasticsearch-0.19.11\config\ansj\userLibrary.dic,UserDict Time:765
[2012-11-14 11:29:17,062][INFO ][ansj-analyzer ] [Dict Loading] F:\java\es\elasticsearch-0.19.11\config\ansj\userLibrary.dic,UserDict Time:922
[2012-11-14 11:29:17,890][INFO ][ansj-analyzer ] [Dict Loading] F:\java\es\elasticsearch-0.19.11\config\ansj\userLibrary.dic,UserDict Time:828
[2012-11-14 11:29:18,140][INFO ][gateway ] [226-ES-11] recovered [2] indices into cluster_state
[2012-11-14 11:29:18,406][DEBUG][river.mongodb ] [226-ES-11] [mongodb][mpagerscn] Prefix: [226-ES-11] [mongodb][mpagerscn] - name: river.mongodb
[2012-11-14 11:29:18,406][INFO ][river.mongodb ] [226-ES-11] [mongodb][mpagerscn] Using mongodb server(s): host [192.168.1.117], port [10000]
[2012-11-14 11:29:18,406][INFO ][river.mongodb ] [226-ES-11] [mongodb][mpagerscn] starting mongodb stream: options: secondaryreadpreference [false], gridfs [false], filter [proddb], db [pagerscn], indexing to [pagerscn]/[{}]
[2012-11-14 11:29:18,640][INFO ][river.mongodb ] [226-ES-11] [mongodb][mpagerscn] No known previous slurping time for this collection
[2012-11-14 11:29:18,734][DEBUG][river.mongodb ] [226-ES-11] [mongodb][mpagerscn] Insert operation - id: i - contains attachment: 5091ce58cd8027a03ec6873f
[2012-11-14 11:29:18,734][DEBUG][river.mongodb ] [226-ES-11] [mongodb][mpagerscn] Insert operation - id: i - contains attachment: 5091ce58cd8027a03ec68740

For the above test, I'm using the same mongo jdbc driver (mongo-2.9.3.jar). And I've also tried river with elasticsearch-river-mongodb-1.4.0-SNAPSHOT.jar directly, and it works fine.

The log info directly using 1.4.0 driver:

[2012-11-14 11:41:39,078][INFO ][ansj-analyzer ] [Dict Loading] F:\java\es\elasticsearch-0.19.11\config\ansj\userLibrary.dic,UserDict Time:875
[2012-11-14 11:41:39,921][INFO ][ansj-analyzer ] [Dict Loading] F:\java\es\elasticsearch-0.19.11\config\ansj\userLibrary.dic,UserDict Time:843
[2012-11-14 11:41:39,921][INFO ][cluster.metadata ] [226-ES-11] [pagerscn] creating index, cause [api], shards [3]/[0], mappings [pagerscn]
[2012-11-14 11:41:40,187][INFO ][cluster.metadata ] [226-ES-11] [_river] update_mapping mpagerscn
[2012-11-14 11:41:40,203][INFO ][river.mongodb ] [226-ES-11] [mongodb][mpagerscn] No known previous slurping time for this collection
[2012-11-14 11:41:40,609][DEBUG][river.mongodb ] [226-ES-11] [mongodb][mpagerscn] Insert operation - id: i - contains attachment: 5091ce58cd8027a03ec6873f
[2012-11-14 11:41:40,609][DEBUG][river.mongodb ] [226-ES-11] [mongodb][mpagerscn] Insert operation - id: i - contains attachment: 5091ce58cd8027a03ec68740
[2012-11-14 11:41:40,609][DEBUG][river.mongodb ] [226-ES-11] [mongodb][mpagerscn] Insert operation - id: i - contains attachment: 5091ce58cd8027a03ec68741

I'm sure there's no difference with my mapping. River mapping configuration:

{
_index: _river
_type: mpagerscn
_id: _meta
_version: 1
_score: 1
_source: {
type: mongodb
mongodb: {
host: 192.168.1.117
port: 10000
db: proddb
collection: wddata_cn_qklw
}
index: {
name: pagerscn
type: pagerscn
bulk_size: 50000
bulk_timeout: 30
throttle_size: 100000
}
}
}

I was wondering whether it's some code changed that made this issue (IndexShardMissingException) happen.

Thanks,
Spancer

from elasticsearch-river-mongodb.

Hi Spancer,

I do not believe IndexShardMissingException is generated by the new version
of the river. No code changes has been done in this area between 1.4.0 and
1.6.0-SNAPSHOT
Do you see anything else that could cause this exception in your
environment?
Are you using the same river setting for both version?
I have notice the value of throttle_size (100000) I think it might be high
I would try with a smaller value 5000 or 10000 which should help for your
issue.

Thanks,
Richard.

On Tue, Nov 13, 2012 at 10:47 PM, spancer [email protected] wrote:

Hi Richard,

Thanks for the concern on this topic.

For this issue, I used the latest file (
https://github.com/downloads/richardwilly98/elasticsearch-river-mongodb/elasticsearch-river-mongodb-1.6.0-SNAPSHOT-plugin.zip),
but after I built up the river, it just hang there, no data indiced to ES.
And I got IndexShardMissingException.

Log info (using elasticsearch-river-mongodb-1.6.0-SNAPSHOT.jar) :

[2012-11-14 11:22:23,125][INFO ][ansj-analyzer ] [Dict Loading]
F:\java\es\elasticsearch-0.19.11\config\ansj\userLibrary.dic,UserDict
Time:922
[2012-11-14 11:22:23,953][INFO ][ansj-analyzer ] [Dict Loading]
F:\java\es\elasticsearch-0.19.11\config\ansj\userLibrary.dic,UserDict
Time:828
[2012-11-14 11:22:23,968][INFO ][cluster.metadata ] [226-ES-11] [pagerscn]
creating index, cause [api], shards [3]/[0], mappings [pagerscn]
[2012-11-14 11:22:24,109][DEBUG][action.admin.indices.status] [226-ES-11]
[pagerscn][1], node[eMigkQC9QBKwmw15y73bPA], [P], s[INITIALIZING]: Failed
to execute [
org.elasticsearch.action.admin.indices.status.IndicesStatusRequest@133baddhttps://github.com/org.elasticsearch.action.admin.indices.status.IndicesStatusRequest/elasticsearch-river-mongodb/commit/133badd
]
org.elasticsearch.index.IndexShardMissingException: [pagerscn][1] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.status.TransportIndicesStatusAction.shardOperation(TransportIndicesStatusAction.java:153)
at
org.elasticsearch.action.admin.indices.status.TransportIndicesStatusAction.shardOperation(TransportIndicesStatusAction.java:59)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-11-14 11:22:24,109][DEBUG][action.admin.indices.status] [226-ES-11]
[pagerscn][2], node[eMigkQC9QBKwmw15y73bPA], [P], s[INITIALIZING]: Failed
to execute [
org.elasticsearch.action.admin.indices.status.IndicesStatusRequest@133baddhttps://github.com/org.elasticsearch.action.admin.indices.status.IndicesStatusRequest/elasticsearch-river-mongodb/commit/133badd
]
org.elasticsearch.index.IndexShardMissingException: [pagerscn][2] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.status.TransportIndicesStatusAction.shardOperation(TransportIndicesStatusAction.java:153)
at
org.elasticsearch.action.admin.indices.status.TransportIndicesStatusAction.shardOperation(TransportIndicesStatusAction.java:59)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-11-14 11:22:24,468][INFO ][river.mongodb ] [226-ES-11]
[mongodb][mpagerscn] No known previous slurping time for this collection
[2012-11-14 11:22:24,468][DEBUG][river.mongodb ] [226-ES-11]
[mongodb][mpagerscn] Using filter: { "$and" : [ { "ns" :
"proddb.wddata_cn_qklw"}]}
[2012-11-14 11:22:24,578][INFO ][cluster.metadata ] [226-ES-11] [_river]
update_mapping mpagerscn http://dynamic

However, when I stopped ES service, changed the 1.6.0 river jar to
elasticsearch-river-mongodb-1.4.0-SNAPSHOT.jar, and restart the ES service,
it works. Data polled in and indeced successfully.

The new log info :

[2012-11-14 11:29:07,562][WARN ][bootstrap ] jvm uses the client vm, make
sure to run java with the server vm for best performance by adding -serverto the command line
[2012-11-14 11:29:07,640][INFO ][node ] [226-ES-11] {0.19.11}[20200]:
initializing ...
[2012-11-14 11:29:07,671][INFO ][plugins ] [226-ES-11] loaded
[river-mongodb, mapper-attachments, analysis-ansj, jdbc-river], sites []
[2012-11-14 11:29:09,234][INFO ][node ] [226-ES-11] {0.19.11}[20200]:
initialized
[2012-11-14 11:29:09,234][INFO ][node ] [226-ES-11] {0.19.11}[20200]:
starting ...
[2012-11-14 11:29:09,312][INFO ][transport ] [226-ES-11] bound_address
{inet[/0.0.0.0:9300]}, publish_address {inet[/192.168.1.226:9300]}
[2012-11-14 11:29:14,359][INFO ][cluster.service ] [226-ES-11] new_master
[226-ES-11][cZVOSDaPQyCZVxiTtA824A][inet[/192.168.1.226:9300]]{master=true},
reason: zen-disco-join (elected_as_master)
[2012-11-14 11:29:14,375][INFO ][discovery ] [226-ES-11]
testcluster/cZVOSDaPQyCZVxiTtA824A
[2012-11-14 11:29:14,390][INFO ][http ] [226-ES-11] bound_address
{inet[/0.0.0.0:9200]}, publish_address {inet[/192.168.1.226:9200]}
[2012-11-14 11:29:14,390][INFO ][node ] [226-ES-11] {0.19.11}[20200]:
started
[2012-11-14 11:29:15,328][INFO ][ansj-analyzer ] [Dict Loading]
F:\java\es\elasticsearch-0.19.11\config\ansj\userLibrary.dic,UserDict
Time:860
[2012-11-14 11:29:16,093][INFO ][ansj-analyzer ] [Dict Loading]
F:\java\es\elasticsearch-0.19.11\config\ansj\userLibrary.dic,UserDict
Time:765
[2012-11-14 11:29:17,062][INFO ][ansj-analyzer ] [Dict Loading]
F:\java\es\elasticsearch-0.19.11\config\ansj\userLibrary.dic,UserDict
Time:922
[2012-11-14 11:29:17,890][INFO ][ansj-analyzer ] [Dict Loading]
F:\java\es\elasticsearch-0.19.11\config\ansj\userLibrary.dic,UserDict
Time:828
[2012-11-14 11:29:18,140][INFO ][gateway ] [226-ES-11] recovered [2]
indices into cluster_state
[2012-11-14 11:29:18,406][DEBUG][river.mongodb ] [226-ES-11]
[mongodb][mpagerscn] Prefix: [226-ES-11] [mongodb][mpagerscn] - name:
river.mongodb
[2012-11-14 11:29:18,406][INFO ][river.mongodb ] [226-ES-11]
[mongodb][mpagerscn] Using mongodb server(s): host [192.168.1.117], port
[10000]
[2012-11-14 11:29:18,406][INFO ][river.mongodb ] [226-ES-11]
[mongodb][mpagerscn] starting mongodb stream: options:
secondaryreadpreference [false], gridfs [false], filter [proddb], db
[pagerscn], indexing to [pagerscn]/[{}]
[2012-11-14 11:29:18,640][INFO ][river.mongodb ] [226-ES-11]
[mongodb][mpagerscn] No known previous slurping time for this collection
[2012-11-14 11:29:18,734][DEBUG][river.mongodb ] [226-ES-11]
[mongodb][mpagerscn] Insert operation - id: i - contains attachment:
5091ce58cd8027a03ec6873f
[2012-11-14 11:29:18,734][DEBUG][river.mongodb ] [226-ES-11]
[mongodb][mpagerscn] Insert operation - id: i - contains attachment:
5091ce58cd8027a03ec68740

For the above test, I'm using the same mongo jdbc driver
(mongo-2.9.3.jar). And I've also tried river with
elasticsearch-river-mongodb-1.4.0-SNAPSHOT.jar directly, and it works fine.

The log info directly using 1.4.0 driver:

[2012-11-14 11:41:39,078][INFO ][ansj-analyzer ] [Dict Loading]
F:\java\es\elasticsearch-0.19.11\config\ansj\userLibrary.dic,UserDict
Time:875
[2012-11-14 11:41:39,921][INFO ][ansj-analyzer ] [Dict Loading]
F:\java\es\elasticsearch-0.19.11\config\ansj\userLibrary.dic,UserDict
Time:843
[2012-11-14 11:41:39,921][INFO ][cluster.metadata ] [226-ES-11] [pagerscn]
creating index, cause [api], shards [3]/[0], mappings [pagerscn]
[2012-11-14 11:41:40,187][INFO ][cluster.metadata ] [226-ES-11] [_river]
update_mapping mpagerscn http://dynamic
[2012-11-14 11:41:40,203][INFO ][river.mongodb ] [226-ES-11]
[mongodb][mpagerscn] No known previous slurping time for this collection
[2012-11-14 11:41:40,609][DEBUG][river.mongodb ] [226-ES-11]
[mongodb][mpagerscn] Insert operation - id: i - contains attachment:
5091ce58cd8027a03ec6873f
[2012-11-14 11:41:40,609][DEBUG][river.mongodb ] [226-ES-11]
[mongodb][mpagerscn] Insert operation - id: i - contains attachment:
5091ce58cd8027a03ec68740
[2012-11-14 11:41:40,609][DEBUG][river.mongodb ] [226-ES-11]
[mongodb][mpagerscn] Insert operation - id: i - contains attachment:
5091ce58cd8027a03ec68741

I'm sure there's no difference with my mapping. River mapping
configuration:

{
_index: _river
_type: mpagerscn
_id: _meta
_version: 1
_score: 1
_source: {
type: mongodb
mongodb: {
host: 192.168.1.117
port: 10000
db: proddb
collection: wddata_cn_qklw
}
index: {
name: pagerscn
type: pagerscn
bulk_size: 50000
bulk_timeout: 30
throttle_size: 100000
}
}
}

I was wondering whether it's some code changed that made this issue
(IndexShardMissingException) happen.

Thanks,
Spancer

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/15#issuecomment-10354267.

from elasticsearch-river-mongodb.

Hi Richard,

It's weird. The 1.6 still doesn't work successfully. However, while using 1.4 instead, it works fine...
I'm now concerning on #41.

This makes me a big headache. Is there any possibility that my mongodb version make this happening? (mongo version 2.2.0).

Wishes,
Spancer

from elasticsearch-river-mongodb.

Hi Richard,

It's really weird, while I created a new collection in mongodb, the river runs well, and data were fetched and

indexed. But tables/collection built previously can't river successfully. No error or exception happens. Just shows

' No known previous slurping time for this collection ', which also given out while river runs successfully when

using river plugin version 1.4...

Have you any idea on this?

Thanks,
Spancer

from elasticsearch-river-mongodb.

Hi Spancer,

How large is your oplog.rs collection?

This is a capped collection and river is monitoring it so if the data from
the previous collection have been overwritten in oplog.rs the rive will not
be able to index them. That's the most likely reason of your issue I see.

Thanks,
Richard.

On Mon, Dec 3, 2012 at 2:00 AM, spancer [email protected] wrote:

Hi Richard,

It's really weird, while I created a new collection in mongodb, the river
runs well, and data were fetched and

indexed. But tables/collection built previously can't river successfully.
No error or exception happens. Just

' No known previous slurping time for this collection ', which also given
out while river runs successfully when

using river plugin version 1.4...

Have you any idea on this?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/15#issuecomment-10943306.

from elasticsearch-river-mongodb.

Spancer,

What MongoDB are you using?

Thanks,
Richard.

Sent via BlackBerry Bold

-----Original Message-----
From: Richard Louapre [email protected]
Date: Mon, 3 Dec 2012 06:35:35
To: richardwilly98/[email protected]
Cc: richardwilly98/elasticsearch-river-mongodbelasticsearch-river-mongodb@noreply.github.com
Subject: Re: [elasticsearch-river-mongodb] No known previous slurping time (#15)

Hi Spancer,

How large is your oplog.rs collection?

This is a capped collection and river is monitoring it so if the data from
the previous collection have been overwritten in oplog.rs the rive will not
be able to index them. That's the most likely reason of your issue I see.

Thanks,
Richard.

On Mon, Dec 3, 2012 at 2:00 AM, spancer [email protected] wrote:

Hi Richard,

It's really weird, while I created a new collection in mongodb, the river
runs well, and data were fetched and

indexed. But tables/collection built previously can't river successfully.
No error or exception happens. Just

' No known previous slurping time for this collection ', which also given
out while river runs successfully when

using river plugin version 1.4...

Have you any idea on this?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/15#issuecomment-10943306.

from elasticsearch-river-mongodb.

Hi Richard,

Thanks for the reply.

I'm using mongodb 2.2.0, Centos 6, 64bits system, and I checked my oplog.rs, it holds a size of 63.3MB with
44362 records.

Thanks,
Spancer

from elasticsearch-river-mongodb.

Hi Spancer,

I believe the entries maybe not be in oplog.rs for the other collection.
Can you run this query (replace mydb and mycollection) by the correct values.
db.oplog.rs.count({"ns": "mydb.mycollection", "op": "i"})
Please compare with a count query directly in your collection.

Thanks,
Richard.

from elasticsearch-river-mongodb.

I am seeing the same issue with the latest versions (Mongo: 2.4.1 and ElasticSearch: 0.20.6). db.oplog.rs.count({"ns": "testmongo.person", "op": "i"}) returns 0 even right after I dropped the person collection and recreated with the following command:
use testmongo
var p = {firstName: "John", lastName: "Doe"}
db.person.save(p)

Any updates on this? Thanks.

from elasticsearch-river-mongodb.

Hi,

If oplog.rs collection is not populated the replicaset has not been correctly created.

Please review it. It has nothing to do with the river.

Thanks,
Richard.

Sent via BlackBerry by AT&T

-----Original Message-----
From: yuganghu [email protected]
Date: Thu, 04 Apr 2013 09:24:58
To: richardwilly98/elasticsearch-river-mongodbelasticsearch-river-mongodb@noreply.github.com
Reply-To: richardwilly98/elasticsearch-river-mongodb [email protected]
Subject: Re: [elasticsearch-river-mongodb] No known previous slurping time
(#15)

I am seeing the same issue with the latest versions (Mongo: 2.4.1 and ElasticSearch: 0.20.6). db.oplog.rs.count({"ns": "testmongo.person", "op": "i"}) returns 0 even right after I dropped the person collection and recreated with the following command:
use testmongo
var p = {firstName: "John", lastName: "Doe"}
db.person.save(p)

Any updates on this? Thanks.

Reply to this email directly or view it on GitHub:
#15 (comment)

from elasticsearch-river-mongodb.

Hi,

I have commited an example on how to create a simple replica set in MongoDB [1].
It has been tested in Windows environment.
Make sure to define MONGO_HOME environment variable and change DB_PATH in mongo-node-*.bat.
Then execute start-replicaset.bat. Once all MongoDB instances have been started run create-replicaset.bat.

[1] - https://github.com/richardwilly98/elasticsearch-river-mongodb/tree/master/resources/issues/64/simple-replicaset

Thanks,
Richard.

from elasticsearch-river-mongodb.

Hi Richard,

When I am trying to create Elastic Search River with mongodb with the help of this command

plugin -install richardwilly98/elasticsearch-river-mongodb/1.4.0

I am getting following error

-> Installing richardwilly98/elasticsearch-river-mongodb/1.4.0...
Trying http://download.elasticsearch.org/richardwilly98/elasticsearch-river-mongodb/elasticsearch-river-mongodb-1.4.0.zip...
Trying http://search.maven.org/remotecontent?filepath=richardwilly98/elasticsearch-river-mongodb/1.4.0/elasticsearch-river-mongodb-1.4.0.zip...
Trying https://oss.sonatype.org/service/local/repositories/releases/content/richardwilly98/elasticsearch-river-mongodb/1.4.0/elasticsearch-river-mongodb-1.4.0.zip...
Trying https://github.com/richardwilly98/elasticsearch-river-mongodb/zipball/v1.4.0... (assuming site plugin)
Failed to install richardwilly98/elasticsearch-river-mongodb/1.4.0, reason: failed to download out of all possible locations..., use -verbose to get detailed information.

Please help me out.

from elasticsearch-river-mongodb.

Hi,

Is there a specific reason you are trying to upload version 1.4.0?
The readme on the homepage [1] provide the versions of ES and MongoDB support.
Look at the "Install Guide" section of the wiki [2] .

[1] - https://github.com/richardwilly98/elasticsearch-river-mongodb
[2] - https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki#install-guide

Thanks,
Richard.

from elasticsearch-river-mongodb.