When a service is deleted via the REST API, it's an active removal instead of passive like in the case of an automatic expiration. For that reason, Eureka should lower the renews threshold value so that an alert state will be resolved.

Not sure if we push jars to maven central or not, but artifacts.netflix.com (internal) has a 1.1.92 version that isn't tagged here. Thought I'd file incase this indicates a build process bug.

During self-preservation mode, eureka clients try to download the full registry instead of deltas. There does not seem to be any apparent benefit for this - we should continue with fetching the delta fetch and download full registry only if the clients cache and the server information gets out of sync.

https://github.com/Netflix/eureka/blob/master/eureka-client/src/main/java/com/netflix/appinfo/AmazonInfo.java#L118

autoBuild seems to have an algorithm that retries awsMetaDataRetries in case of error. I expected to see a break from the while (numOfRetries-- > 0) upon success (where it does result.metadata.put(key.getName(), value);). Instead in a version of this we copied and changed, we see it just try every query awsMetaDataRetries times.

We could submit a patch for this when we submit our similar class. However, I wanted to double check that I understood the code right and there is really a issue here.

Inside netflix we wish to add a grouping criteria for different application under an umbrella. For this purpose eureka needs to publish the application group which then can be used by other clients to do any logic based on the application group.

The application group can be specified via a property "namespace" + "appGroup" where namespace by default is "eureka." and hence the property eureka.appGroup can be used to specify the application group. Alternatively, in absence of this property, eureka will also look at a java environment property "NETFLIX_APP_GROUP" to infer the application group. The requirement of this environment property is purely due to internal requirements.

When eureka server startups, it tries to find the unused EIPS based on the list from it neighboring nodes.

If that communication fails for some reason, multiple eureka servers will bind to the same EIP there by snatching it from one another.

AWS does not prevent reassociation in EC2-classic. A better way to handle this edge-case scenario is to find if the an instance id is associated with an elastic IP and if so skip that binding.

Getting the following issue when trying to start up eureka server:

Dec 31, 2012 4:02:51 AM org.apache.catalina.core.StandardContext listenerStart
SEVERE: Exception sending context initialized event to listener instance of class com.netflix.eureka.EurekaBootStrap
java.lang.RuntimeException: Cannot bootstrap eureka server :
at com.netflix.eureka.EurekaBootStrap.contextInitialized(EurekaBootStrap.java:120)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4791)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5285)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:633)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:968)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1646)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.lang.ExceptionInInitializerError
at com.netflix.eureka.EurekaBootStrap.contextInitialized(EurekaBootStrap.java:100)
... 14 more
Caused by: java.lang.IllegalStateException: java.lang.NullPointerException
at com.netflix.eureka.PeerAwareInstanceRegistry.setupPeerEurekaNodes(PeerAwareInstanceRegistry.java:197)
at com.netflix.eureka.PeerAwareInstanceRegistry.init(PeerAwareInstanceRegistry.java:150)
at com.netflix.eureka.PeerAwareInstanceRegistry.(PeerAwareInstanceRegistry.java:138)
at com.netflix.eureka.PeerAwareInstanceRegistry.(PeerAwareInstanceRegistry.java:124)
... 15 more
Caused by: java.lang.NullPointerException
at com.netflix.logging.messaging.MessageBatcher.(MessageBatcher.java:136)
at com.netflix.logging.messaging.BatcherFactory.createBatcher(BatcherFactory.java:70)
at com.netflix.eureka.cluster.PeerEurekaNode.getBatcher(PeerEurekaNode.java:577)
at com.netflix.eureka.cluster.PeerEurekaNode.(PeerEurekaNode.java:88)
at com.netflix.eureka.PeerAwareInstanceRegistry.updatePeerEurekaNodes(PeerAwareInstanceRegistry.java:213)
at com.netflix.eureka.PeerAwareInstanceRegistry.setupPeerEurekaNodes(PeerAwareInstanceRegistry.java:181)
... 18 more

Full catalina.out: https://gist.github.com/4417295

The apache HTTP client, used by eureka for making HTTP requests to eureka servers does not have any metrics for connection pool. We should atleast be adding the following:

1. Available connections per host.
2. Idle connections per host.
3. Total connections per host.
4. In flight requests per host.

This code:

does 7 network calls for every AmazonInfo.autoBuild() call which gets triggered by default every 30 seconds by InstanceInfoReplicator thread in DiscoveryClient.

We can optimize this to only fetch the public hostname & IP which can only change after an instance has started. We can cache everything else.

From the description of the jersey-bundle artifact:

Such a bundle is only intended for developers that do not use Maven's dependency system.

Consider specifying in POM/gradle build script exactly which Jersey artifacts are required. The inclusion of the bundle artifact leads to issues with managing transitive dependency conflicts of different Jersey versions.

The changes in issue #29 enabled eureka client to fetch instances from a remote region (if available with the server) if configured via a property. This behavior is determined at client initialization. This will cause applications to restart whenever a change in behavior is required.
We should make this decision dynamically everytime we refresh the client's cache. This will lead to clearing up the local cache and fetching entire registry whenever the property changes.

The console currently shows the number per application, but does not have the total numbers for the entire registry.

Original thread is at https://groups.google.com/forum/#!topic/eureka_netflix/Rntihgo1i50

My usecase is that I'd like to have 1 service with multiple instances running on the same host. Since they are the same service, they should have the same app name. And since they are on the same host, they have the same hostname. The problem is that in a non-AWS datacenter, the instanceID is populated by the hostname. Same app name and same instanceId leads to overwriting in the service registry. I would appreciate it if the EurekaInstanceConfig interface had a getId() method that I can implement.

Thank you.

Latency and cost may sometimes force applications to be run on one AWS region and then fallback to applications for other region.

There should be support for region fallback if an application does not exist in one region if the fallback region is configured.

karthik-

any thought to using jetty (similar to exhibitor) or embedded tomcat (similar to asgard) for an easier standalone startup? this would simplify the deploy instructions a bit as it would remove the dependency on a separate tomcat install.

additionally, this would allow additional CLI params like --port to allow eureka to startup on a custom port.

i really like how jordan built - and documented - his exhibitor CLI: https://github.com/Netflix/exhibitor/wiki/Running-Exhibitor

just a thought - this would address one of the items that we discussed at the recent Netflix OSS meetup. specifically, the fact that my instructions to deploy eureka on a custom port are very involved: https://github.com/cfregly/fluxcapacitor#build-and-deploy-eureka-server

lemme know what you think.

thanks!

-chris

1. Eureka client tries to talk to a server in the same zone and fails over to the next server predictably for all clients in that zone. This could put a lot of pressure on the server that all these clients are failing over. These clients should distribute the load to other servers in the case of failover. This can be achieved by choosing a different failover server depending on the client.

2. Currently when there are 2 servers in the same zone, the eureka client always chooses the same server every time. The second server is not utilized until failover. This restricts the ability to loadbalance requests across multiple servers in the same zone.

The current version of AmazonInfo prepends the namespace parameter to property suffixes such as mt.read_timeout. Since other Netflix projects such as Zuul use the namespace eureka, the resulting properties look like eurekamt.read_timeout instead of eureka.mt.read_timeout.

I'm filing this against Eureka instead of Zuul because I think Eureka should be adding a . when there is a namespace -- I'm happy to submit a PR that does this and also falls back on the old behavior -- but I can understand if the response is instead that Zuul's usage should be changed.

Currently, EIPManager.getCandidateEIP() will prefer EIPs attached to unreachable instances first if they appear in the available EIP list first.
This can be optimized more by first choosing the EIP which is not attached to any instance (as per other peer nodes registry info) at all.

Since eureka-client directly depends on ribbon-httpclient version 0.3.10, it should use jersey-apache-client4 version 1.11 which is compatible with it instead of version 1.8.

Here is the output of maven enforcer plugin which points to the discrepancy:
Dependency convergence error for com.sun.jersey.contribs:jersey-apache-client4:1.11 paths to dependency are:

  +-com.netflix.eureka:eureka-client:1.1.127
    +-com.netflix.ribbon:ribbon-httpclient:0.3.10
      +-com.sun.jersey.contribs:jersey-apache-client4:1.11
and
  +-com.netflix.eureka:eureka-client:1.1.127
    +-com.sun.jersey.contribs:jersey-apache-client4:1.8

If the renewals falls below the threshold for a legitimate reason, it would be very helpful to have a method to reset the threshold to clear the alert state. As it is now, the only way to reset it is to restart Eureka.

Hi there,
We are trying to run Eureka in a solely VPC environment and don't have the need for EIP registration. Instead we are considering adding functionality to have Eureka register itself directly in Route53 as discovery.us-west-2c.xyz.com, or similar. That way we can make Eureka part of an ASG, avoid middle-tier ELBs, and use round-robin DNS once we need more than one Eureka node per AZ.

Before we go through the effort, what is your guys' thinking around VPC deployments of Eureka and would you be interested in us committing this back?

Cheers
Nils

The fix for issue #29 derives the region for an availability zone based on a property.
This same information is available for netflix environment in DNS entry.
Hence, if available (i.e. eureka client is configured to fetch service urls from DNS) we should use the DNS for this mapping. Also, the mapping must be refreshed everytime we fetch data from the eureka server.

Eureka has two instance states, one as overridden status and other as actual instance status published by the instance.

The override is generally used to manage instance state from an administration tool like asgard. It will be handy to have an API TO remove all overridden state which will start honoring instance status as published by the instance itself.

As discussed in this thread: https://groups.google.com/forum/?fromgroups#!topic/eureka_netflix/ddBO6ostCp0 usage of StatsTimer in this class: https://github.com/Netflix/eureka/blob/master/eureka-client/src/main/java/com/netflix/discovery/shared/EurekaJerseyClient.java#L270 causes memory leak because of misconfiguration of StatsTimer.

In this particular case, we do not need the percentile times provided by StatsTimer over BasicTimer. So, changing to BasicTimer should resolve this issue.

On Amazon, when eureka is brought down, it releases the EIP address. Amazon does not reassign a new public ip address to the instance for atleast a couple of minutes once the EIP is released. Restarting the eureka server during this time causes the EIP binding to fail the first time because the public IP is null. Here's the stack trace:

2013-08-20 15:05:55,365 ERROR com.netflix.eureka.EurekaBootStrap:266 [localhost-startStop-1] [bindEIP] Cannot bind to EIP
java.lang.NullPointerException
at com.netflix.eureka.util.EIPManager.getCandidateEIP(EIPManager.java:196)
at com.netflix.eureka.util.EIPManager.bindToEIP(EIPManager.java:114)
at com.netflix.eureka.EurekaBootStrap.bindEIP(EurekaBootStrap.java:263)
at com.netflix.eureka.EurekaBootStrap.handleEIPbinding(EurekaBootStrap.java:222)

It would be nice to allow the binding to go ahead if the current ip is null.

The first implementation of fetching instances from a remote registry was done in a way that iff there are no instances of an application in the local region, then we get instances from remote region transparently, without the caller knowing it.
The above worked for the usecase inside Netflix initially. However, the flaw is that there was no way to fetch instances from a remote region if there were some instances of the same app in the local region. This will be prohibitive in an active-active multiple region setup.
Issue #29 provided a new way of fetching instances from a remote region.

This issue here is created to phase out the older behavior in a controlled way.

I ran into some issues when I tried to setup multiple zones with eureka. According to the wiki, the txt records should be separated with ','. However, it seems like the code is using the whitespace character?
Wiki page I am referring to:
https://github.com/Netflix/eureka/wiki/Configuring-Eureka-in-AWS-Cloud

But on this line it splits using whitespace:
https://github.com/Netflix/eureka/blob/master/eureka-client/src/main/java/com/netflix/discovery/DiscoveryClient.java#L1366

This would be fine if it was as simple as a code->wiki mismatch. But when there is a whitespace within a txt record it returns it as "eu-west1a eu-west1b", instead of with comma as eu-west1a,eu-west1b (hard to illustrate but in the first case it returns a string containing actual quotation marks). So when it splits it winds up with:

• "eu-west1a
• eu-west1b"

Which of course causes problems.

Am I the only one who ran into this issue? Like the way that the TXT-records and automatic assigning of EIPs but it's very fragile to have only one eureka server.

I would like Eureka to support multiple services hosted within the same JVM,
which would allow the eureka-client to register all of them instead of the current 1 app per process model (The singleton design prevents me from creating multiple DiscoveryClients in the same JVM).

I would like to know if others may require such feature, in order to decide how to proceed with bridging this gap.

Thanks,
Elad.

Currently discovery client determines the region of an instance by looking up the availability zone into the set of known availability zone -> region mapping. This mapping is determined via DNS or properties.
However, in case, when there is no valid information in this mapping for an availability zone, it puts this instance into the local bucket. We can have one fallback which can see if the availability zone follows a format like: us-east-1c where us-east-1 is the region name and "c" is the zone. If so, we can see if the resulting name is a valid, known region, if so, use the same.

I am trying to configure two Eureka servers on a pair of machines on a LAN to verify inter-server communication is configured correctly.

It would be extremely helpful to have a sample configuration for this configuration so that I can verify everything is setup/working correctly.

As far as I can tell, the only way to update metadata for a service is to re-register the service. It would be super handy for me if there existed a PUT endpoint similar to the {...}/status?value= endpoint that could be used to update the metadata, in part or in whole.

Currently eureka provides a way to get registry information from a remote aws region.
However, the behavior of this is not controllable by the user.
The instances from the remote region is added to the local registry only if the application, the instance belongs to, is not already available locally.

This change should allow users to retrieve instances for a specific region (via the eureka client).

The following setup would be required to achieve this:

Eureka Server

1. The eureka server which will get data from a remote region must create a property: eureka.remoteRegionUrlsWithName with the content as key value pairs separated by a comma. The key is the name of the region and value is the http(s) url to the eureka server in the remote region. This will deprecate the existing property: eureka.remoteRegionUrls which does not have any insight into which region the URL belongs to.
2. Optionally provide a whitelist of applications, per region (or global) which will restrict only a certain set of applications to be retrieved from a remote region. The retrieval here is referred to as the API view of the remote registry. The server internally will fetch the entire remote registry. The whitelist follows the property name: eureka.remoteRegion.[region_name].appWhiteList, where [region_name] can either be the string "global" or the name of the region (must match with what is provided in 1 above). The value is a comma separated list of app names. The global list is only consulted when there is no whitelist for the specific region. In absence of any whitelist all instances from the remote region will be retrieved.

Client

1. Define a property: eureka.fetchRemoteRegionsRegistry with the value as comma separated list of regions from which this client should fetch the registry information. This obviously requires that the server actually is pulling data from those regions.
2. For each remote region that the client need to fetch, define a property: "eureka.[region_name].availabilityZones" containing a comma separated list of all availability zones in the region. This is to reverse map, the availability zones (available as instance metadata) in eureka server registry response. Since, this mapping is static, it adds little value to add the extra parameter region in all the instance information. We can possibly infer this from availability zone, i.e. us-west-1c is in region us-west-1 but this is more or less to externalize this logic rather than bake in the code.
3. Use the methods in DiscoveryClient to get instances for a specific region. In absence of a region, the client will fetch instances from the local region only.

Caveat

If the client choose this mode of fetching instances, it can not go back to the previous way of getting remote region instances i.e. only applications that are not available locally will be fetched from remote region without the client knowing about it.

Hello,

I can see the metrics exposed through Servo and JMX successfully. How do i integrate CloudWatch though? Where should i provide AWS credentials? Are these a standard accessKey and secretKey combination? For the server configuration, i can see the accessKey and secretKey configuration parameters. Are these used for CloudWatch integration? If this is the case, i do not see these parameters for the client configuration. Is CloudWatch available only to server?

Eureka client on shutdown does not cancel all the scheduled tasks it has scheduled.
This may create a race-condition under which, the client may re-register with the server even after shutdown.

The race-condition lies in the Instance Info replication scheduled task which if there is a healthcheck handler registered with eureka, will re-register with the eureka server.

The fix is to cancel all the scheduled tasks at shutdown.

Discovery client while querying from the backup registry, in the event of all eureka servers not being available, does not honor remote regions requested.

Currently, if the eureka server is configured to fetch data from a remote region registry but fails to initialize the same, the error is ignored.
This results in unexpected behavior of the server without any indication of what is wrong. We should fail fast in such a scenario.

When starting Eureka in a dev environment with no other Eureka servers available, the startup times takes between 3 and 5 minutes. I was that setting the following parameters would bypass this:

eureka.enableSelfPreservation=false
eureka.registration.enabled=false

However, it still is taking 3-5 minutes. Is there something else that needs to be set for this to work?

I'm going through the code to just crush as many generic CheckStyles issues as I can, and noticed that a lot of lines longer than 120 characters would be fine if we went to Java 7 for the diamond operator. We'd also pick up a bunch of other Java 7 enhancements. How do you feel about requiring Java 7?

is there a way to avoid the following code snippet used to bridge the archaius and eureka configs:

https://github.com/cfregly/fluxcapacitor/blob/master/flux-edge/src/main/java/com/fluxcapacitor/edge/server/EdgeServer.java

i understand the desire to let eureka run standalone without requiring an archaius configuration, but is there a best-of-both-worlds approach where archaius will be used if these properties don't exist, maybe?

lemme know.

-chris

RemoteRegionRegistry:

tries to access response object which will be null if the remote discovery is not reachable.

Currently a server can only offer services on 2 ports when integrated with Ribbon since announced addresses are limited to port/secureport.

Ideally an arbitrary number of ports could be supported. Or at the very least 3, one each for http, https with client auth, https without client auth.

We have been extending the eureka server to add additional capabilities to it. But we would like to be able to do this without actually making any changes to the eureka code itself.
One nice feature that would help us if to have the ability to register a callback function that should be invoked whenever an instance is evicted by eureka (once it's become non responsive). That would allow us to keep our code base independent of the eureka code base.

Add a callback to allow for listeners to be notified when the eureka client gets an update in it's poller.

I'm managing a connection pool to a set of instances in a cluster. I'd like to be able to react in a timely manner to status changes of those instances. E.G. mark-down, mark-up, instance termination and metadata changes.

My current strategy is to poll the client periodically, but that introduces even more latency on top of the latency introduced by the polling thread.

Eureka uses jersey apache http client for eureka server communications. In order to cleanup connections, it relies on closing the ClientResponse object returned by jersey. In cases, when there is an unexpected error (like OOM) during reading of response, there is no ClientResponse object that is returned by jersey. In such a case, there happens to be a connection leak in the apache http client connection pool.

karthik-

i'm working on an end-to-end reference app that uses a lot of the Netflix OSS components and i'm having trouble integrating the latest build of eureka.

you can find the reference app here, btw: https://github.com/cfregly/refapp

is there any known issue around depending on an older, unreleased version (0.4.13) of servo-core?

to reproduce, just uncomment out the eureka-core dependency in refapp-core/pom.xml and do a maven build.

another piece of info: eureka-client by itself seems to be fine. it's only when either eureka-server, eureka-core, or eureka-resources are introduced.

i tried the latest SNAPSHOT from source and it's giving the same error.

i also tried the usual rm -rf .m2/repository directory, etc.

i wanted to check with you before i spend more time digging into the dependencies.

any guidance would be helpful.

thanks!

-chris

The connection error is silently ignored.
This should not be the case because this will be cause an issue of eureka not showing up instances from the other region.

Feature request:

Add option for variable validation in Eureka Client, for example validation options for what "application" field can look like. An example of an optional validation could look like something provided by Frigga already:

https://github.com/Netflix/frigga/blob/master/src/main/java/com/netflix/frigga/NameValidation.java

When I attempt to POST to /v2/apps/APPID, if the Content-Type request header is "application/json; charset=utf-8", the application throws an exception. If the header is simply "application/json", it works as expected. This is on 1.1.72.

Here is the stack trace for the failing scenario:

java.lang.NullPointerException
    com.netflix.eureka.PeerAwareInstanceRegistry.register(PeerAwareInstanceRegistry.java:444)
    com.netflix.eureka.resources.ApplicationResource.addInstance(ApplicationResource.java:124)
    sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    java.lang.reflect.Method.invoke(Unknown Source)
    com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
    com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$VoidOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:167)
    com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
    com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
    com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:134)
    com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
    com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
    com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
    com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
    com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
    com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
    com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
    com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
    com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
    com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
    com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:895)
    com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:843)
    com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:804)
    com.netflix.eureka.StatusFilter.doFilter(StatusFilter.java:68)

Currently, the replication pool is shared across all peer nodes and hence if one of the peers go down it severely affects the ability to replicate to other nodes.

Move the renewals to a separate pool for its peer node so that a peer going down does not affect other peers in an adverse way.

For applications that do not make any outbound calls, do not need to fetch the discovery registry information which can have significant memory overhead depending on how many instances are in the registry.