3.0.3 upgrade fails about aws-iam-identity-center-extensions HOT 14 CLOSED

Hi @allquixotic ,

Thank you for raising this issue. This is occurring because we introduced an additional config parameter in 3.0.3 called UpgradeFromVersionLessThanV303 and this is not present in your config file. Could you update your config file to match the config\env.yaml sample file in terms of keys and retry please? Also, could you set the UpgradeFromVersionLessThanV303 parameter to true as you are upgrading from earlier versions and the account assignment data would need to be upgraded to the new convention i.e. move from dot(.) to percent(%).

Additionally, we would like to understand if our release notes (for v3.0.3 - https://github.com/aws-samples/aws-sso-extensions-for-enterprise/releases/tag/v3.0.3) are easy to understand and follow. Could you help us with any feedback that would help us improve/change how we write our release notes?

Thank you,
Leela

from aws-iam-identity-center-extensions.

Additionally, we would like to understand if our release notes (for v3.0.3 - https://github.com/aws-samples/aws-sso-extensions-for-enterprise/releases/tag/v3.0.3) are easy to understand and follow. Could you help us with any feedback that would help us improve/change how we write our release notes?

The release notes are clear. I did not see them before, which is the reason I didn't follow them. Had I seen them, this issue probably wouldn't have occurred. Thank you.

from aws-iam-identity-center-extensions.

That fixed everything except for FileAsset21, which failed with

Action execution failed
Error calling startBuild: Cannot have more than 20 builds in queue for the account (Service:  AWSCodeBuild; Status Code: 400; Error Code: AccountLimitExceededException

from aws-iam-identity-center-extensions.

Also, env-aws-sso-extensions-for-enterprise-upgradeToV303Stack.Prepare and env-aws-sso-extensions-for-enterprise-upgradeToV303Stack.Deploy are "Didn't Run" status. That is strange.

As a result, my links_data are still in the old format. Maybe they didn't run because the asset failed?

from aws-iam-identity-center-extensions.

That fixed everything except for FileAsset21, which failed with

Action execution failed
Error calling startBuild: Cannot have more than 20 builds in queue for the account (Service:  AWSCodeBuild; Status Code: 400; Error Code: AccountLimitExceededException

This is due to your account hitting the concurrent build quota limit as described here.
Trigger for reaching the concurrent builds quota limit is due to the number of lambda functions we have deployed as part of the solution. CDK pipeline deployment does the packaging of each lambda function (transpiled TS code along with dependencies) as a separate and concurrent build job in the build stage of the pipeline. With the new lambda functions we added as part of v3.0.3(around format changing), you are hitting this no.
Once the quota limit increase self service request is complete on your account, you could then proceed with release change on the pipeline and this should successfully pass the build stage.

from aws-iam-identity-center-extensions.

Also, env-aws-sso-extensions-for-enterprise-upgradeToV303Stack.Prepare and env-aws-sso-extensions-for-enterprise-upgradeToV303Stack.Deploy are "Didn't Run" status. That is strange.

As a result, my links_data are still in the old format. Maybe they didn't run because the asset failed?

Yep, because the build stage has failed the pipeline effectively stopped. Once you have redeployed the pipeline by clicking on release change after the quota increase is in place, this would trigger the stack deployment and your links_data would then be upgraded to the new format

from aws-iam-identity-center-extensions.

My AWS CodeBuild Concurrently Running Builds quota limit is at the default of 60 according to the Service Quotas dashboard. Why does the error say 20? Is this the correct quota to request?

Also, is there any way to modify the SSOEx solution so that it doesn't fire off so many concurrent builds, so that a quota increase isn't necessary?

from aws-iam-identity-center-extensions.

My AWS CodeBuild Concurrently Running Builds quota limit is at the default of 60 according to the Service Quotas dashboard. Why does the error say 20? Is this the correct quota to request?

Also, is there any way to modify the SSOEx solution so that it doesn't fire off so many concurrent builds, so that a quota increase isn't necessary?

Cannot have more than x builds in the queue is a symptom of exceeding the concurrent build quota, so we are looking at the correct quota limit increase. Could you retry by releasing the change on the pipeline and see if the build stage goes through ?
Regarding the parallel build jobs trigger, it speeds up the pipeline deployment to parallelise any build job where feasible and this is one of those scenarios where it would be more optimal to parallelise them instead of running the build jobs serially. Additionally, given that we use CDK's native pipeline construct, this is the default behaviour i.e. the build stage is defined by the construct itself. We could look at an escape hatch route to control this, but for the reasons mentioned previously, we would prefer this to be parallel.

from aws-iam-identity-center-extensions.

Releasing the change caused the build to fail again, only with a different file artifact. I've opened a support ticket with AWS business support under the category of CodeBuild to have them identify which limit is being hit. It doesn't make sense that the error says 20 but my quota (the default quota for all AWS accounts) is 60, so I'm asking them to clarify.

from aws-iam-identity-center-extensions.

AWS support said "For some platforms and compute types, the default is 20." and linked me to https://docs.aws.amazon.com/codebuild/latest/userguide/limits.html#service-quotas

I still don't understand where this number of 20 is coming from. It isn't listed anywhere in the documentation. The other question is, if the default limit is 60, what number should I request in my quota increase to get SSOEx to build successfully when it seems only to require about 25 concurrent builds (out of the "60" I have available)?

None of this makes any sense still. I'm completely confused.

from aws-iam-identity-center-extensions.

AWS support said "For some platforms and compute types, the default is 20." and linked me to https://docs.aws.amazon.com/codebuild/latest/userguide/limits.html#service-quotas

I still don't understand where this number of 20 is coming from. It isn't listed anywhere in the documentation. The other question is, if the default limit is 60, what number should I request in my quota increase to get SSOEx to build successfully when it seems only to require about 25 concurrent builds (out of the "60" I have available)?

None of this makes any sense still. I'm completely confused.

I am confused as well @allquixotic . Few of our team mates had this exact issue and raising the quota limit for conurrent builds fixed it for us. At this point of time, other than asking you to follow up with AWS support, unfortunately I don't have an answer that helps with the issue.

from aws-iam-identity-center-extensions.

Hmm... OK. So for the SSO Extensions project, I can see two possible enhancements (one, the other, or both) helping with this:

(1) Document this limitation in the README.md;

(2) Look into an escape hatch in CDK that would allow us to keep the concurrency of builds to 20 or less, so the issue is not encountered.

I certainly wouldn't want SSOEx to shift to a "serial" concurrency model (one build at a time), but I do hope there is some concurrency factor maximum that can be requested, akin to old school make -j20 :) Hopefully CDK / CodePipeline / CodeBuild has anticipated this fairly typical use case of their platform, when it doesn't take a very complicated solution to hit such a limit. I've read much of the SSOEx code, and while it is very elegant and uses best practices, it isn't the most complicated thing in the world... and we are already hitting unfriendly limits on AWS.

If not... this may need to be a feature request upstream, either to CDK or to the CodeBuild service team. And it feels like a bug anyway that the error is about 20 concurrent builds but we can't find a service limit anywhere that says 20.

Some definitely weird stuff going on here. I've submitted a service limit request to raise the concurrent builds to 80, but I feel there are multiple opportunities for improvement here, not only with SSO Extensions, but with AWS service and with CDK.

Thanks for your insight.

from aws-iam-identity-center-extensions.

After receiving the quota increase for concurrent builds, everything in CodePipeline/CodeBuild works, and I no longer receive the error emails when trying to add users/groups that already exist via SCIM API.

from aws-iam-identity-center-extensions.

Hi @allquixotic ,

@jmejco did some research on this and observed that the speed advantage we would gain through parallel builds for asset publishing is being nullified by the code build instance provisioning lag across the 20 concurrent build jobs that are being triggered for packaging all lambda assets. As this is not adding any advantage from a speed perspective, we've reverted this behaviour i.e. disable the parallel trigger and instead force the pipeline to specifically use one code build job for packaging all lambda assets. This behaviour is now the default from v3.1.1.

from aws-iam-identity-center-extensions.