reconnect to mongos and elasticsearch about monstache HOT 20 CLOSED

hi thanks for reporting this. do you happen to use plugins either javascript or golang? Usually, when I kill the server and try to access it I get the error, io.EOF, which prints as "EOF". I can try to detect these scenarios and refresh the connection. However, this looks different in that the error is "Closed Explicitly" which sounds to me like maybe the use of a plugin is involved in the root cause. I think somewhere session.Close() might be getting called. I should be copying the session when it goes into "user code" so that calling Close on it does not close the main connection.

from monstache.

yes, i get the error "EOF", sometimes.
ERROR 2018/01/30 08:41:48 Unable to save timestamp: EOF
ERROR 2018/01/30 08:41:48 EOF
I was trying emulating connection drop to elasticsearch and mongodb, what's I see in logs:
ERROR 2018/01/30 08:41:42 elastic: http://es-test:9200 is dead
ERROR 2018/01/30 08:41:42 elastic: bulk processor "monstache" failed: Post http://es-test:9200/_bulk: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
ERROR 2018/01/30 08:41:42 elastic: all 1 nodes marked as dead; resurrecting them to prevent deadlock
ERROR 2018/01/30 08:41:42 elastic: bulk processor "monstache" failed: no available connection: no Elasticsearch node available
but after restore net, I don't see traffic on ports elasticsearch and mongodb from monstache service.

from monstache.

one more strange thing, if I start monstache with parameters "verbose", I got see this:

"Workers": [
{
"Queued": 1000,
"LastDuration": 1513000000
},
{
"Queued": 1000,
"LastDuration": 492000000
},
{
"Queued": 0,
"LastDuration": 720000000
},
{
"Queued": 0,
"LastDuration": 889000000
},
{
"Queued": 0,
"LastDuration": 635000000
},
{
"Queued": 0,
"LastDuration": 1184000000
},
{
"Queued": 245,
"LastDuration": 213000000
},
{
"Queued": 1000,
"LastDuration": 134000000
},
{
"Queued": 1000,
"LastDuration": 1874000000
},
{
"Queued": 1000,
"LastDuration": 1666000000
}
]

and moustache will be stopping, but if I press any key:

"Workers": [
{
"Queued": 0,
"LastDuration": 1000000
},
{
"Queued": 18,
"LastDuration": 0
},
{
"Queued": 0,
"LastDuration": 0
},
{
"Queued": 0,
"LastDuration": 0
},
{
"Queued": 0,
"LastDuration": 0
},
{
"Queued": 0,
"LastDuration": 0
},
{
"Queued": 0,
"LastDuration": 0
},
{
"Queued": 0,
"LastDuration": 0
},
{
"Queued": 0,
"LastDuration": 0
},
{
"Queued": 0,
"LastDuration": 0
}
]

from monstache.

The workers output looks like the stats which are described here

As far as the connection loss, can you please do a fresh install from latest in master using

go get -u github.com/rwynn/gtm
go get -u github.com/rwynn/monstache

I've made some changes just now which may help with reconnection.

from monstache.

I have one more questions about logs:
ERROR 2018/01/30 11:47:24 TypeError
what's is mean?
and one more questions about /stats:
Failed": 2901
what's is mean?

from monstache.

Regarding the TypeError message, I have not seen that before. And the message doesn't give us a lot of context to go on. Did you configure custom javascript or go code?

Regarding stats, Failed: 2901 means that 2,901 requests to Elasticsearch have failed from the current Monstache process. These would be bulk requests so each request may contain more than one document for indexing. The stats reset to 0 when you restart the Monstache service.

from monstache.

Yes, I use custom javascript. One more question, sorry, but I have use this service in my project and I need to know - how to get more info about TypeError?

from monstache.

If you remove the Javascript do you still get the TypeError? If you only get the error with the javascript, can you post it here so that maybe I can determine the problem?

I think a TypeError would occur when, for example, you called a method on null, or called a method that doesn't exist. e.g. null.anything() or "abc".doSomething().
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/TypeError

Also, I discovered more about the flood of error messages in the logs when Elasticsearch is stopped. I created a PR for the golang elastic client library for this issue. olivere/elastic#698

from monstache.

hi @rwynn
I sometimes get the same issue and I see @olivere already merged the PR.
Any chance you can build a new version with that included
I would then also update the docker image

from monstache.

@cmosgh No, I didn't? I think I need to look into the details first.

from monstache.

@olivere
Thought since tests were passing you did. My bad

it looked green :)

from monstache.

thank you for your help

from monstache.

@cmosgh I just pushed a new 3.6.0 release that you can use to update the docker image. This includes some fixes for lost connections while we wait for the elastic PR to be reviewed. I switched the import over to my fork temporarily.

from monstache.

Reconnecting to MongoDB and Elasticsearch has been fixed in 3.6.2. Thanks!

from monstache.

Does this fix also apply to direct read? I've been getting

Error performing direct reads of collections: read tcp : i/o timeout

and syncing just stops completely.

from monstache.

The logic is here https://github.com/rwynn/gtm/blob/master/gtm.go. I don’t think that error is handled. Bunch of other io errors are though. I need to find where that error is defined in the standard library. Couldn’t find it here with the others https://golang.org/pkg/io/#pkg-variables

from monstache.

@benan789 can you please re-download the latest release (3.x for ES2-5, 4.X for ES6+) of Monstache and try this again? I have reworked the tailing library that Monstache uses, rebuilt the binaries, and run some tests. I updated this library do a periodic ping after ANY type of error encountered (these are reads so there really isn't much that can happen besides broken connections). Hopefully, it won't exit early from now on.

from monstache.

Still having issue doing a full sync of about 10mil documents, suspecting the meta collection.

#44

from monstache.

What is the error from the log with the latest monstache code?

from monstache.

It's the same error actually. Maybe it didn't stop completely but just didn't update in a while and I thought it stopped lol.

from monstache.