[BUG] type-safe-api: intermittent ENOTEMPTY in CodeBuild about aws-pdk HOT 5 OPEN

Having the same issue here.

from aws-pdk.

Hi I am sorry you are running into this issue.

• Can you please send me what your build spec looks like and also what image you are using?
• How often are you seeing this issue?
• Ideally seeing full logs would help me debug further.
• To resolve this error are you simply restarting the build action?

from aws-pdk.

I was able to reproduce this locally by running this command in parallel:

npx --yes -p @aws/pdk type-safe-api.clean-openapi-generated-code --code-path .

What is happening is:

Process 1 creates a new directory and starts installing the @aws/pdk dependencies in to the /root/.npm/_npx/ folder
Process 2 comes in and tries to do the same thing which is leading to a corruption in this directory. This also explains why it fails on CI but not locally as in CI the /root/.npm/_npx/ will be empty wheras locally it will most likely already have the packages installed.

I am not sure why this is presenting more now, however this would have been an issue for quite some time as apparently npx and dlx are not concurrency-safe. What is interesting is this issue is only present if the /root/.npm/_npx/ folder does not have @aws/pdk and if the folder you are currently running in doesn't have PDK installed (which it should). The resolution path should looks as follows:

1.) Check all node_modules folders starting with the current folder, traversing upward to see if @aws/pdk@$AWS_PDK_VERSION exists.
2.) Fall back to global node_modules located in /root/.npm/_npx/.

I would have assumed that 1) should be resolved as the build command would be called after performing an install. In the event it falls back to 2), this will create a new folder and is where the issue you are seeing is occuring.

Can you please confirm that you have @aws/pdk installed within your local node_modules and it matches the version listed in your $AWS_PDK_VERSION (you can find this in the .projen/tasks.json file)?

As a stop gap there are two options:

Option 1 - Preload the package into the npx cache (recommended)

Add this before your build command: npx --yes -p @aws/pdk@<YOUR_PDK_VERSION> pdk --version.

Replace <YOUR_PDK_VERSION> with the same AWS_PDK_VERSION found within .projen/tasks.json. This will ensure that the package is already present in the npx cache and when the scripts are executed within the build command, it won't try to download the package in parallel (which is where the problem stems).

Option 2 - Force your build to run serially

Append the --parallel 1 flag to your build command to force each package to build in parallel.

Note: This will slow down your build.

I would advise you to try Option 1 and report back whether this helps resolve your issue. If so I will go ahead and raise a PR to incorporate Option 1 into the PDK so you don't have to explicitly perform that step.

from aws-pdk.

To add to this, there may be other stages in the build process that rely on just in time packages like for example if using pnpm, there is a pnpm dlx command that is equivalent to npx which will run into the same problems. We may need to perform this precaching step for these instances also i.e: pnpm --package=@aws/pdk dlx pdk --version.

from aws-pdk.

The issue seems to be related to the release of the latest version of pnpm, version 9. After pinning the version to version 8, the build succeeds.

from aws-pdk.