Upgrading to 3.x results in broken ES client about monstache HOT 12 CLOSED

Also on investigating the ES logs we see the following:

[2017-06-21T13:55:39,970][WARN ][c.f.s.s.h.n.SearchGuardSSLNettyHttpServerTransport] [3rZ6PTp] Someone (/127.0.0.1:45334) speaks http plaintext instead of ssl, will close the channel

I think that shows that the issue is the new Elastic Go driver not communicating with an TLS enabled cluster.

from monstache.

@Crispy1975, check the new client options in 3.0. Mostly the same but elasticsearch url config is changed to accomodate golang driver change in monstache.

https://rwynn.github.io/monstache-site/options/#elasticsearch-urls

Notice change to elasticsearch-urls and it is an array now. It might have tried default when this did not match - localhost:9200.

Also, I believe the sniffing option of the elastic client has been turned off since 3.0. So it should just attempt to use the array given.

https://github.com/rwynn/monstache/blob/master/monstache.go#L218

from monstache.

No dice. I had already tried setting the URLs to an array, but did it again to be sure I hadn't made a mistake, this is what I have:

elasticsearch-url = [ "https://es0:9200", "https://es1:9200", "https://es2:9200" ]

The hosts are in the /etc/hosts on the box monstache is running from and I've checked connectivity, can also curl ES no problem. So network is up and fine, but still getting the same error.

I also compiled a version with sniffing set the true and that also resulted in the error, this still looks related to the TLS transport, as watching the ES logs I see monstache making contact over the network, but as per the error below ES thinks it's communicating in plain text and not SSL.

[2017-06-22T09:02:41,193][WARN ][c.f.s.s.h.n.SearchGuardSSLNettyHttpServerTransport] [3rZ6PTp] Someone (/127.0.0.1:52316) speaks http plaintext instead of ssl, will close the channel

It says 127.0.0.1 as I am testing this actually on one of the ES nodes.

from monstache.

Can you try adding the s to the end of elastic-url in config? So elasticsearch-urls. Keep the array value as that is correct.

from monstache.

ERROR 2017/06/22 14:07:07 elastic: http://10.128.3.1:9200 is dead
ERROR 2017/06/22 14:07:07 elastic: http://10.128.3.2:9200 is dead
ERROR 2017/06/22 14:07:07 elastic: http://10.128.3.3:9200 is dead
ERROR 2017/06/22 14:07:07 Unable to create elasticsearch client: no active connection found: no Elasticsearch node available
panic: Unable to create elasticsearch client: no active connection found: no Elasticsearch node available

goroutine 1 [running]:
log.(*Logger).Panicf(0xc420016c80, 0xd27181, 0x29, 0xc420529d40, 0x1, 0x1)
        /usr/local/go/src/log/log.go:215 +0xdb
main.main()
        /home/crispy/godev/src/github.com/rwynn/monstache/monstache.go:1423 +0x47c

Note it's translating the hosts from https to http.

from monstache.

Looks like the driver does not auto detect the scheme. Can you do a local build of Monstache and set the scheme on the Client to https?
https://github.com/olivere/elastic/blob/release-branch.v5/client.go

from monstache.

Also try reverting the sniff enable you changed

from monstache.

@rwynn perfect, that build with the client set to https is working for us (also reverted the sniffing change). Will put the build through it's paces in our staging environments and see how things go under load there. 👍

from monstache.

@Crispy1975, just pushed a new release to fix the https scheme problem. Please let me know if you find any issues with this release.

from monstache.

@rwynn thanks, well test it first thing in the morning and feed back.

from monstache.

@Crispy1975, is it OK if I close this issue? Any improvements under load from 3.x?

from monstache.

@rwynn yes I believe we are seeing improvements. We have also upgraded our infrastructure that MongoDB is running on which gives a boost but I think we are seeing better throughput from monstache also. All good. 🎉

from monstache.