ECS EC2 Deployment - Cloudformation Time Out error about yelb HOT 24 CLOSED

Yeah I will test the cfn in another account with default vpc , and get back to you.

from yelb.

Hey we found the problem, DNS Hostname was disabled on the VPC. After enabling it, we launched the template and it ran successfully! Thanks a lot for the help. really appreciate it. We will test this for EC2 now.

from yelb.

Closing this.

from yelb.

Hey there. Yes the CFN templates assumes the cluster already exists. If you use Fargate you don't need much else but if you use EC2 you need to deploy 1+ EC2 Container Instances and attach them to the cluster to provide capacity. This is outside of the goal of this CFN template. I should perhaps make it clear in the README.

from yelb.

Thanks. I have created a 2 container instances of m5.large and registered with the cluster.

The appserver, DB and redis services launched successfully. But the UI service is erroring out with exit code 1.

It is unable to find the yelb-appserver in the nginx conf looks like. Is this related to the startup script for UI?

from yelb.

Mh... let me test this.

from yelb.

Weird. I just tested it and it works fine with instances deployed to the default VPC. It looks like name resolution isn't working properly. Is it possible that your VPC is configured in a way that disables DNS resolution? This is usually a reason why this scenario fails. Also, does this work if you try to deploy to Fargate (Vs EC2)?

from yelb.

I just checked, the vpc has DNS resolution enabled. Only DNS hostnames are disabled.

from yelb.

That in theory should not be a problem (but I have not tested it). Can you confirm that both CloudMap and Route53 have a namespace and hosted zone for the arbitrary domain you have picked in the CloudFormation template to support name resolution? And that all records in both the namespace and hosted zones match and are consistent?

For example, if you pick as a domain name (like I did) yelb-ec2.local you should have a CloudMap namespace called yelb-ec2.local and a R53 hosted zone called yelb-ec2-local.

To further troubleshoot this it would interesting to see if you can deploy the template using Fargate (to rule out any EC2 related misconfiguration) and possibly with a new VPC setup with all default values and public subnets for simplicity.

from yelb.

Sure, will test the script for Fargate and get back to you. Thanks a lot for the immediate response, we greatly appreciate your help!

from yelb.

Absolutely. I am sorry you are going through this. I'd love to help more.

from yelb.

Hey,

We launched the Fargate cluster but got the same error - the UI service fails in connecting to the app server

from yelb.

Mh... there is something going on here. First I assume you are using the stock CFN template provided in the repo (no changes)? What is the domain you set as an input? Can you show me how the R53 zone associated to that domain looks like? (please make sure that you do not capture confidential information). Can you also show me how the yelb-appserver service instances in the Cloud Map namespace that corresponds to the domain? Like for me these are the two tasks in the yelb-appserver Cloud Map service.

from yelb.

Hey,

Yes we are using the stock CFN template for ECS, no changes.

This is how the configuration looks:

1. Route 53:
   

2. Cloud Map Services:
   

3. Cloud Map App Server Service: Only one task is listed

4. Cloud Map UI Server Service: Unhealthy

from yelb.

Yeah one yelb-appserver is fine because it depends how many you have set in the input of CFN (default is one). More worrying is the ui that shows unknown in CloudMap and apparently doesn't show up in the hosted zone in R53? Does all look good in the CFN stack?

This is how my R53 hosted zone looks like (I have 3 UI tasks and 2 appserver tasks in my deployment).

from yelb.

oh, on our side the R53 doe not have any entries for UI.

this is the script we are using, no modifications - https://github.com/mreferre/yelb/blob/master/deployments/platformdeployment/AWS/ECS/cfn/yelb-cloudformation-ECS-deployment.yaml

The CFN is stuck at this stage and in Cloud Map there are no service instances listed for UI.

Not sure if there is a limitation issue.

Thanks again!

from yelb.

Mh... if you are using Fargate makes sure that your limits allows you to deploy all your tasks. You can check the Quotas in the AWS console (search for quotas and then look for Fargate as the service).

If you go into the ECS Console and you explore the yelb-ui service do you see anything in particular? Especially in the Health and metrics, Configurations and tasks, Deployment and events, etc.

from yelb.

Also, I doubt that the Fargate quota is the problem because 1) you showed the container logs above so that task has been deployed and 2) you seemed to have the same problem when deploying to EC2. In addition to exploring the ECS console (see above) does the CFN stack say anything interesting? What are theResources that are still pending the deployment? Does the Events tab say anything interesting?

from yelb.

Hey,
I have checked health and metrics of ui server.

Configuration and task

deployment and events:

And only ui server is pending , not running in the cfn stack.

from yelb.

This is weird. There is something not working in the name resolution. The yelb-ui is particularly sensitive to this at startup because of the nginx proxy configuration (if it fails to connect to http://yelb-appserver it will exit/abort. The yelb-appserver is not as sensitive at startup but I am pretty sure that if you hit its apis to query the data it will fail too at run-time because it won't be able to connect to redis and the db.

Usually, based on my experience, these problems are caused by VPC/accounts misconfigurations (typically DNS resolution). I think the next step would be to test this in another account with a default VPC. Do you happen to have such an account for a quick test?

from yelb.

Gosh. I knew it was DNS. It's always DNS!

DNS *hostnames* fooled me because I have associated it to EC2 and the problem was clearly not EC2. We should have Enabled that early on! Well, happy it's working now. We learned something new.

from yelb.

Hey,
Sorry to trouble you again, but in ec2-ecs deployment I m getting error. Before deploying the cfn, I have created two m5large instance, I have given ecs credential in the ec2 but it failed.

from yelb.

Mh... looks like a problem with the setup of the instances and the ECS agent. Not sure if the role has anything to do with it. The new ECS console allows you to create the ASG + ECS container instances as part of cluster creation. My suggestion would be to start there for a test.

from yelb.

Ok will check and let you know.

from yelb.