Failing to fetch registry about discovery HOT 10 CLOSED

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/161556933

The labels on this github issue will be updated when the story is started.

from discovery.

Hi @ReidThompson,

• What version of .NET are you targeting?
• What ASP.NET version are you using?
• Which Steeltoe service discovery package & version are you using?
• Is the service registry being provisioned by Spring Cloud Services?
• Is the registry instance healthy?
• Does the app succeed in fetching the registry if you restart it?

from discovery.

I'm targetting .NetCore v2.1
I'm using ASPNetCore v2.1.5
I'm using the following Steeltoe packages:

• Pivotal.Discovery.ClientCore v2.1.1
• Steeltoe.CircuitBreaker.Hystrix.MetricsStreamCore v2.1.1
• Steeltoe.CircuitBreaker.HystrixCore v2.1.1

My assumption is that the service registry is being provisioned by Spring Cloud Services. I had created this particular registry for a couple of Java apps, and it works as intended. However, for clarity's sake, I'd be happy to check somewhere to give you a better answer. Is this something I would find in the pcf gui? Or by running cf services {{service-registry}}?

The registry manage page is telling me that it's up, so my assumption is that it's healthy. And as I said a moment ago, it works for my java apps.

The app does not succeed in fetching the registry if I restart it.

To elaborate on what's happening, I've successfully gotten it to fetch the registry only a handful of times, and then when it goes to renew, it tells me the fetch has failed. I've tried restarting the app in the PCF gui, and I've tried rerunning cf push. In both cases, I am not guaranteed to fetch the registry successfully. I would say it happens around 25%.

from discovery.

Thanks for that info. I just wanted to make sure the instance of Eureka is healthy and not something out of the ordinary ~ if you did the provisioning, but don't know of anything special about it then it probably is the SCS version, and if it works for your Java apps then there's probably nothing wrong with it.

What stack are you deploying the app to? (eg: windows2012R2, windows2016, cflinuxfs2, cflinuxfs3)
Can you confirm that DNS is working in the cell you're deploying to?
Have you tried deploying one of the Steeltoe Discovery samples onto your infrastructure to make sure that it isn't an environmental issue?

No such device or address typically means either DNS has failed, or you've made a request to a bad server address. Configuring the log level for Steeltoe to Debug or Trace could provide some additional information.

from discovery.

I am deploying to cflinuxfs2; I'm told we'll be switching to cflinuxfs3 at some point, but I wasn't given a date.

I have not personally tried one of the Steeltoe Discovery samples, I'll try that next, but it's worth noting that I did comb over every relavant file and tried the exact same configurations (Program.cs, Startup.cs, appsettings.json, and the relevant ServiceCommand class).

Below is the most recent logs, after configuring the log level to Debug. You can see that it initially works, but when the renew happens, it fails.

2018-10-30T09:23:34.480-04:00 [APP/PROC/WEB/0] [OUT] FetchFullRegistry returned: OK, Applications[Application[Name=SERVICE-CONSUMER,Instances=Instance[InstanceId=service-consumer.mypcf.com:40f6a64f-187f-4b4b-6736-33c1,HostName=service-consumer.mypcf.com,IpAddr=TRUNCATED_IP,Status=UP,IsUnsecurePortEnabled=True,Port=80,IsSecurePortEnabled=True,SecurePort=443,VipAddress=service-consumer,SecureVipAddress=service-consumer,ActionType=ADDED],]Application[Name=SERVICE-GENERATOR,Instances=Instance[InstanceId=service-generation.mypcf.com:78916739-9693-4009-4d94-2d04,HostName=message-generation.mypcf.com,IpAddr=TRUNCATED_IP,Status=UP,IsUnsecurePortEnabled=True,Port=80,IsSecurePortEnabled=True,SecurePort=443,VipAddress=service-generator,SecureVipAddress=service-generator,ActionType=ADDED],]Application[Name=EUREKA-SERVER,Instances=Instance[InstanceId=eureka-a51cd99a-1bd9-4883-92fb-71abb8c0d4d1-node1.mypcf.com:625d47f4-347a-4f6e-75f5-230f,HostName=eureka-a51cd99a-1bd9-4883-92fb-71abb8c0d4d1-node1.mypcf.com,IpAddr=TRUNCATED_IP,Status=UP,IsUnsecurePortEnabled=False,Port=8080,IsSecurePortEnabled=True,SecurePort=443,VipAddress=eureka-server,SecureVipAddress=eureka-server,ActionType=ADDED],Instance[InstanceId=eureka-a51cd99a-1bd9-4883-92fb-71abb8c0d4d1-node2.mypcf.com:82152383-25b6-40df-52bb-a8e1,HostName=eureka-a51cd99a-1bd9-4883-92fb-71abb8c0d4d1-node2.mypcf.com,IpAddr=TRUNCATED_IP,Status=UP,IsUnsecurePortEnabled=False,Port=8080,IsSecurePortEnabled=True,SecurePort=443,VipAddress=eureka-server,SecureVipAddress=eureka-server,ActionType=ADDED],Instance[InstanceId=eureka-a51cd99a-1bd9-4883-92fb-71abb8c0d4d1-node4.mypcf.com:99071fc6-77c6-497e-7130-57d2,HostName=eureka-a51cd99a-1bd9-4883-92fb-71abb8c0d4d1-node4mypcf.com,IpAddr=TRUNCATED_IP,Status=UP,IsUnsecurePortEnabled=False,Port=8080,IsSecurePortEnabled=True,SecurePort=443,VipAddress=eureka-server,SecureVipAddress=eureka-server,ActionType=ADDED],Instance[InstanceId=eureka-a51cd99a-1bd9-4883-92fb-71abb8c0d4d1-node5.mypcfcom:c7dd3c34-49b9-47de-7d85-a614,HostName=eureka-a51cd99a-1bd9-4883-92fb-71abb8c0d4d1-node5.mypcf.com,IpAddr=TRUNCATED_IP,Status=UP,IsUnsecurePortEnabled=False,Port=8080,IsSecurePortEnabled=True,SecurePort=443,VipAddress=eureka-server,SecureVipAddress=eureka-server,ActionType=ADDED],Instance[InstanceId=eureka-a51cd99a-1bd9-4883-92fb-71abb8c0d4d1-node3.mypcf.com:41096447-36a1-45c3-5156-c4f4,HostName=eureka-a51cd99a-1bd9-4883-92fb-71abb8c0d4d1-node3.mypcf.com,IpAddr=TRUNCATED_IP,Status=UP,IsUnsecurePortEnabled=False,Port=8080,IsSecurePortEnabled=True,SecurePort=443,VipAddress=eureka-server,SecureVipAddress=eureka-server,ActionType=ADDED],]]
2018-10-30T09:23:34.481-04:00 [APP/PROC/WEB/0] [OUT] �[40m�[37mdbug�[39m�[22m�[49m: Steeltoe.Discovery.Eureka.DiscoveryClient[0]
2018-10-30T09:23:34.481-04:00 [APP/PROC/WEB/0] [OUT] FetchRegistry succeeded
2018-10-30T09:23:34.577-04:00 [APP/PROC/WEB/0] [OUT] Hosting environment: Development
2018-10-30T09:23:34.577-04:00 [APP/PROC/WEB/0] [OUT] Content root path: /home/vcap/app
2018-10-30T09:23:34.578-04:00 [APP/PROC/WEB/0] [OUT] Now listening on: http://0.0.0.0:8080
2018-10-30T09:23:34.578-04:00 [APP/PROC/WEB/0] [OUT] Application started. Press Ctrl+C to shut down.
2018-10-30T09:23:36.552-04:00 [CELL/0] [OUT] Container became healthy
2018-10-30T09:23:40.669-04:00 [CELL/0] [OUT] Cell ba95d92b-087b-4804-835c-4bf37f84ca24 destroying container for instance b1668f7d-ac3d-441e-590d-6b26
2018-10-30T09:23:40.964-04:00 [CELL/0] [OUT] Cell ba95d92b-087b-4804-835c-4bf37f84ca24 successfully destroyed container for instance b1668f7d-ac3d-441e-590d-6b26
2018-10-30T09:24:04.551-04:00 [APP/PROC/WEB/0] [OUT] �[41m�[30mfail�[39m�[22m�[49m: Steeltoe.Discovery.Eureka.Transport.EurekaHttpClient[0]
2018-10-30T09:24:04.551-04:00 [APP/PROC/WEB/0] [OUT] DoGetApplicationsAsync Failed
2018-10-30T09:24:04.551-04:00 [APP/PROC/WEB/0] [OUT] System.Net.Http.HttpRequestException: No such device or address ---> System.Net.Sockets.SocketException: No such device or address
2018-10-30T09:24:04.551-04:00 [APP/PROC/WEB/0] [OUT] at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
2018-10-30T09:24:04.551-04:00 [APP/PROC/WEB/0] [OUT] --- End of inner exception stack trace ---
2018-10-30T09:24:04.551-04:00 [APP/PROC/WEB/0] [OUT] at System.Threading.Tasks.ValueTask1.get_Result() 2018-10-30T09:24:04.551-04:00 [APP/PROC/WEB/0] [OUT] at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken) 2018-10-30T09:24:04.551-04:00 [APP/PROC/WEB/0] [OUT] at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask1 creationTask)
2018-10-30T09:24:04.551-04:00 [APP/PROC/WEB/0] [OUT] at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
2018-10-30T09:24:04.551-04:00 [APP/PROC/WEB/0] [OUT] at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
2018-10-30T09:24:04.551-04:00 [APP/PROC/WEB/0] [OUT] at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) 2018-10-30T09:24:04.551-04:00 [APP/PROC/WEB/0] [OUT] at Steeltoe.Discovery.Eureka.Transport.EurekaHttpClient.DoGetApplicationsAsync(String path, ISet1 regions)
2018-10-30T09:24:04.556-04:00 [APP/PROC/WEB/0] [OUT] �[41m�[30mfail�[39m�[22m�[49m: Steeltoe.Discovery.Eureka.DiscoveryClient[0]
2018-10-30T09:24:04.556-04:00 [APP/PROC/WEB/0] [OUT] FetchRegistry Failed
2018-10-30T09:24:04.556-04:00 [APP/PROC/WEB/0] [OUT] System.Net.Http.HttpRequestException: No such device or address ---> System.Net.Sockets.SocketException: No such device or address
2018-10-30T09:24:04.556-04:00 [APP/PROC/WEB/0] [OUT] at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
2018-10-30T09:24:04.556-04:00 [APP/PROC/WEB/0] [OUT] --- End of inner exception stack trace ---

from discovery.

Well, two more things to try, in addition to deploying the Steeltoe samples -

1. I pushed a new release of Discovery to the dev feed that logs the url when the request fails, so we can at least rule out any surprises there - the version is '2.2.0-dev-287'
2. Possibly a long shot, but opt out of the SocketsHttpHandler that shipped with .NET Core 2.1 might give us other ideas if the results change

from discovery.

I tried with just adding the new release so I could verify the URL. It is indeed trying to connect my instance of the service registry.

Also, just now I tried adding the method you linked, and it appears to be working. It's been up for about 15 minutes now, and it's continuing to succeed on fetching.

Now, the issue I'm running into is not being able to reach my other .Net service by name. Now that I can successfully fetch the registry, I was able to test against my java service that exists in the same registry. I can consume that one. However, when I try to consume my .Net service, I'm tripping my hystrix circuit breaker. Unfortunately, I must leave the office now. I'll update tomorrow when I'm able to test out the sample apps.

from discovery.

I threw the Fortune Teller service up on our instance of pcf, and it also required the method you linked, and is also always tripping the circuit breaker, despite being told in the logs that the registry was fetched/registered.

from discovery.

Is it fair to say this is a duplicate of #46? it seems there may be a little more digging to do here, it would be nice to consolidate to one thread

from discovery.

Hey Tim,

I was out of office Mon/Tues, so I apologize for my delay in getting back to you. Yes, this issue and #46 are related. Originally when we opened the issues, we weren't sure if the root problem was the same, but as we've worked through it, it appears that that is the case. I will close this issue.

from discovery.