0 Logs being processed. about firehose-to-syslog HOT 11 CLOSED

Hi,

Which version are you using ?
30 000/min is not that big 500 /sec if you have 6 nozzle that's mean around 80 logs by nozzle.
I am not sure if the loggregator send in the Round Robin way to all of 6.

Btw when this happen what cf app youapp give you ? We use a small on disk database so maybe this DB is full..

Even if you capture only "LogMessage" the nozzle receive all traffic, because the filtering is done on the nozzle level.

from firehose-to-syslog.

I'm using version 1.3.1

Looking at the "Log Event Totals" messages, the incoming logs seem to be evenly distributed among the instances. Whenever there is an increase in log volume, all of the events processed by each instance increase, similarly when the volume decreases so do the events processed across all the instances.

While examining the logs it looks like the issue is related to the instance not responding to a ping in time.

Tue Apr 05 2016 08:58:53 GMT-0400 (Eastern Standard Time) [APP] ERR [2016-04-05 12:58:53.564120254 +0000 UTC] Exception occurred! Message: Firehose Error! Details: websocket: close 1008 Client did not respond to ping before keep-alive timeout expired.

Tue Apr 05 2016 08:58:53 GMT-0400 (Eastern Standard Time) [APP] OUT We have processed 0 events from the firehose at wss://doppler.domain.com:443 over the last 9.9 seconds and 77575 total events since startup

#0 running 2016-04-05 08:33:09 AM 5.3% 23.6M of 512M 14.8M of 512M
#1 running 2016-04-05 08:33:09 AM 0.0% 25.8M of 512M 14.8M of 512M
#2 running 2016-04-05 08:33:09 AM 5.0% 21.6M of 512M 14.8M of 512M
#3 running 2016-04-05 08:33:09 AM 14.5% 21.6M of 512M 14.8M of 512M
#4 running 2016-04-05 08:33:09 AM 10.7% 23.6M of 512M 14.8M of 512M
#5 running 2016-04-05 08:33:09 AM 5.0% 19.6M of 512M 14.8M of 512M

After the instances fails to respond to ping before the keep-alive timeout, it's a hit or miss weather it will come back. From the error above, instance 1 started processing logs again within 10 - 15 minutes. However, I've witnessed it where all of the instances went down and stopped processes logs for 24+ hours.

Is there a way to increase the keep-alive timeout?

from firehose-to-syslog.

Thanks.
Seems the noaa lib have been updated, so I will updated firehose-to-syslog too.
How many loggregator / doppler to you have ?

By curiousity do you have any network equipment preventing the keep-alive (firewall)?

from firehose-to-syslog.

In each of the cloud foundry foundations we have 2 Doppler and 2 Loggregator VMs (1 vCPU/ 2GB mem).

There shouldn't be any network equipment preventing the keep-alive. The firehose-to-syslog nozzle is running as an application. From the garden container it's running in to the traffic controller, there's nothing blocking it's path.

from firehose-to-syslog.

Ok, I will try to update the middleware.

from firehose-to-syslog.

@ahahtyler Please compile from this branch feature/update_nooa_keepalive using go1.6

$godep go build

There is a new options

  --fh-keep-alive=25s            Keep Alive duration for the firehose consumer

Let me know

from firehose-to-syslog.

Built and running. Will let it run over the weekend and see if it still times out.

from firehose-to-syslog.

Still happening ???

from firehose-to-syslog.

Sorry for the late reply. Extending the keep alive seems to have done the trick. Thanks for the help!

from firehose-to-syslog.

Ok, I will cut off new release this week.

Thanks

from firehose-to-syslog.

@shinji62 I am not following why this is a keepalive setting in client since after digging into the source code of server side websocket implementation and client side websocket implementation, it appears the keepalive is initiated by server instead of client ? When server doesn't receive Pong response from client, it will report policy violation and timeout error.

--fh-keep-alive is used for websocket client reading packets timeout, it seems not related to keepalive to me.

https://github.com/cloudfoundry-attic/loggregatorlib/blob/master/server/websocket_keepalive.go

Thank you !

from firehose-to-syslog.