BUG - Builds images with sparse files which cannot be copied about cfn-ecr-aws-soci-index-builder HOT 6 OPEN

Hey, thanks for opening an issue! I was able to reproduce the issue with index builder and the SOCI CLI. When you created the index with the CLI did you explicitly set a --min-layer-size? If not, I presume the layer with the sparse file (the last layer) was not lazy loaded since it was under the 10 MiB default min layer size threshold set by the CLI, and therefore the cp did not hang. The index builder sets the min-layer-size to 0 by default, which means that each layer in the image was lazy loaded and therefore susceptible to any SOCI FS bugs. See: #40

We are looking into this issue with sparse files on the SOCI side. We will keep you posted.

from cfn-ecr-aws-soci-index-builder.

Hey, just wanted to provide a quick update. We have root caused the issue to be in one of our dependencies, go-fuse. We have a PR out against them upstream. In the meantime we are actively working on resolving #40, so you can build a working SOCI index for your image with the CFN solution. @bgardner-noggin

from cfn-ecr-aws-soci-index-builder.

#40 (--min-layer-size issue) has now been resolved/merged. You should now be able to re-deploy your CFN stack and have a working SOCi index built for your image. Please let us know if you run into any issues and thanks for your patience. @bgardner-noggin

from cfn-ecr-aws-soci-index-builder.

Hey, sorry for the wait, we had some issues with updating our release process but they are resolved now. Release v0.4.1 is now out, so if you are consuming SOCI via ECS on EC2 you can now pull this release in. If you are using Fargate this fix should already be available to you. Thank you for your patience!

from cfn-ecr-aws-soci-index-builder.

0.4.1 has been released, but this project has not been updated to use it.

I pulled this code, changed go.mod to use 0.4.1 and built the zip using the Dockerfile

However, when I execute the lambda against my ECR repo, it only generates a "Soci index" image in the ECR repo. Beforehand it would generate two items per image

"Soci index"
"Image Index"

I cannot see anything in the log to indicate why it isn't generating the "Image index" though

from cfn-ecr-aws-soci-index-builder.

0.4.1 has been released, but this project has not been updated to use it.

The underlying issue was in the snapshotter, not in any of the build tools like the CLI or index builder, so you should be able to use the same index. If you are using ECS Fargate, you should just be able to deploy your tasks. If you are on ECS on EC2, you can pull the latest version of the snasphotter in. Let us know if you run into any other issues.

I pulled this code, changed go.mod to use 0.4.1 and built the zip using the Dockerfile
However, when I execute the lambda against my ECR repo, it only generates a "Soci index" image in the ECR repo. Beforehand it would generate two items per image

We have a PR out upgrading the index builder to pull the latest version of SOCI (v0.4.1). It seems like we were able to generate the image index, as-well. Once that is in you can re-deploy the solution and retry.

from cfn-ecr-aws-soci-index-builder.