Non-passing members not being removed from Members list about hazelcast-consul-discovery-spi HOT 8 CLOSED

Please post your configuration

from hazelcast-consul-discovery-spi.

Note that the class com.jungo.hazelcast.ConsulDiscoveryStrategyWrapperFactory is just a class that allows me to conditionally enable this discovery based on the embedded "enabled" property since apparently spring doesn't support the enabled tag on the discovery-strategy element itself

           <hz:join>
                <hz:multicast enabled="false"></hz:multicast>
                <hz:tcp-ip enabled="#{ '${hazelcast.consul.enabled}' == 'true' ?  'false': 'true' }">
                    <hz:members>${cluster.membersOf.BOTH_LAYERS}</hz:members>
                </hz:tcp-ip>
                <hz:aws enabled="false"></hz:aws>
                <hz:discovery-strategies>
                  <hz:discovery-strategy discovery-strategy-factory="com.jungo.hazelcast.ConsulDiscoveryStrategyWrapperFactory">
                    <hz:properties>
                      <hz:property name="enabled">#{ '${hazelcast.consul.enabled}' == 'true' ?  'true' : 'false' }</hz:property>
                      <hz:property name="consul-host">localhost</hz:property>
                      <hz:property name="consul-port">8500</hz:property>
                      <hz:property name="consul-service-name">${hazelcast.consul.service}</hz:property>
                      <hz:property name="consul-healthy-only">true</hz:property>
                      <hz:property name="consul-service-tags"></hz:property>
                      <hz:property name="consul-discovery-delay-ms">10000</hz:property>

                      <hz:property name="consul-acl-token"></hz:property>
                      <hz:property name="consul-ssl-enabled">false</hz:property>
                      <hz:property name="consul-ssl-server-cert-file-path"></hz:property>
                      <hz:property name="consul-ssl-server-cert-base64"></hz:property>
                      <hz:property name="consul-ssl-server-hostname-verify">true</hz:property>

                      <hz:property name="consul-registrator">org.bitsofinfo.hazelcast.discovery.consul.DoNothingRegistrator</hz:property>
                    </hz:properties>
                  </hz:discovery-strategy>
                </hz:discovery-strategies>
            </hz:join>

from hazelcast-consul-discovery-spi.

Sorry, obviously didn't mean to close this issue.

from hazelcast-consul-discovery-spi.

Looking at your config and that you are using the DoNothingRegistrator I would assume the following:

a) you have a consul agent running as a sidecar to each hazelcast app

b) You have manually configured each consul agent with manual checks against the local HZ app it is an agent for. (what those checks are I have no idea)

c) each of your consul agents is reporting into a cluster of consul servers

Assuming the above is true, and if you kill one of your consul agents, and you execute a curl http://localhost:8500/v1/health/service/hazelcast?passing above as described, and the HZ node that the "dead" consul agent is monitoring is still listed... then this sounds like a misconfiguration in the way you have consul setup, vs a problem w/ this lib.

The consul-healthy-only attribute is only utilized by this library when it consults the configured consul endpoint to get the list of hazelcast peers it should connect to. That is typically done on HZ startup or a re-initialization at runtime.

Again, something is amiss with the way you have your consul infrastructure setup if your agent dies, and the node it is monitoring is still listed by other consul nodes and never going away. Something in that setup or the health checks you have setup in consul (or lack thereof, or they are not working etc)

from hazelcast-consul-discovery-spi.

Sorry I haven't been clear, but you may have answered my question

http://localhost:8500/v1/health/service/hazelcast?passing

does NOT return the hazelcast service for the node that is disconnected. However, looking at the Members list in the server.log for wildfly (where this code is included) does not show an update reflecting that the node was removed from the Members list. If I deregister hazelcast from consul (which occurs when I stop wildfly), the Members list is updated to remove the node, but if the hazelcast service is not passing it is still in the list. I was expecting that the list would reflect the healthy instances of hazelcast

from hazelcast-consul-discovery-spi.

"However, looking at the Members list in the server.log for wildfly (where this code is included) does not show an update reflecting that the node was removed from the Members list"

For which node? The node monitored by the dead consul agent? Or the other nodes connected to active consul agents?

If you kill the consul agent monitoring one hazelcast node, but that hazelcast node is still up, connected to other hazelcast peers and functioning fine, then yes of course other HZ nodes (peers) would still show it as active and up, as once they are all connected, they have established a network. They won't disconnect from peers just because the "active" list they get from consul is reporting that node as no longer up (and in this case incorrectly reporting so, because the HZ node is up). They will only obey that list in consul when "re-initializing" their list of potential peers to connect to and establish connections with.

Consul is like an address book. You use the address book to determine a location. Once you arrive there you are good as long as the location actually exists. Unless that location disappears off the face of the earth, you stay there because it still exists, regardless of what the address book states (from its perspective). You only reconsult the address book after you leave and need to figure out the address again.

from hazelcast-consul-discovery-spi.

Thanks for the reply! That explains the issue exactly. The problem I'm having is that on another node I'm using consul-template that returns a different list of members than hazelcast is using. I've changed my consul-template to return "any" instance, whether consul considers it healthy or not, and in this use case that is sufficient

from hazelcast-consul-discovery-spi.

@pheitman glad to help, if you find the project useful, please star it!

from hazelcast-consul-discovery-spi.