Amazon Import Post Processor Fails about packer-plugin-amazon HOT 7 OPEN

Hi @jeremymcgee73 thanks for reaching out. Just a heads up this issue moved to its new home hashicorp/packer-plugin-amazon as the Amazon components are now being maintained in their own repository. With that said it is is possible that this is an eventual consistency issue.

Does the resource eventually get created and is reachable on S3?

If it is a timing issue you can workaround this by setting a very high value for AWS_MAX_ATTEMPTS and AWS_POLL_DELAY_SECONDS. If after a long wait time you still run into issues I would double check that you are able to create and read from the respective S3 resources.

from packer-plugin-amazon.

Thanks for the reply! I am trying that now, and will let you know what I find out. That does make sense! I thought about that, but didnt think about setting those to be high.

If that is the problem, would it be worth adding a pause and retry for this?

from packer-plugin-amazon.

I'm not 100% sure yet. But, I am pretty certain this has solved my problem. Usually 1 out of 5 failed.

Thanks!

from packer-plugin-amazon.

Thanks for the quick turn around on testing @jeremymcgee73. If that is working then I would say that the retry logic in place already is working its just that the default values are not high enough, at least for your use case. If this is an issue for other's then it might mean we need to reevaluate the defaults. Keeping this open for now.

To better assist with this issue I would recommend adding the the aws_polling configuration option to your templates to avoid having to set ENV variables each time. This was added to override defaults for Amazon services that were not longer enough for some users. Information on the configuration option can be found at https://www.packer.io/docs/builders/amazon/ebs#polling-configuration

from packer-plugin-amazon.

I actually don't think that solved my problem. The OVA is getting copied to S3, because it remains after the run. The problem is very intermittent, maybe only happens every 10 runs. Let me know how else I can help.

Thanks!

Post-processor failed: Import task import-ami-0f0769d3d5ba527d6 failed with status message: ClientError: The specified S3 resource does not exist. Reason 404 Not Found, error: ResourceNotReady: failed waiting for successful resource state

The settings I'm passing in as ENV vars:
AWS_POLL_DELAY_SECONDS=600
AWS_MAX_ATTEMPTS=100

from packer-plugin-amazon.

I took another look at this, I don't believe the AWS poll intervals will help. I believe this is failing on the initial import step, not on the checks to see if its done. I think maybe this particular error could be caught, and tried again?

I believe it may be the size of the images, that creates the race condition. Maybe if you add a step to your tests that copy a big file to the image(get the total up to 5/6GB), before the post-processor is ran.

from packer-plugin-amazon.

I would recommend adding the the aws_polling configuration option to your templates

Hi @nywilken 👋🏼

Just wondering how to set the aws_polling configuration option on a build that is using the amazon-import post-processor?

We're using the hyperv-iso builder so we have something like the following defined:

source "hyperv-iso" "build" {
  communicator       = "winrm"
  ...
}
...
build {
  description = "Windows 10"
  ...
  provisioner "windows-restart" {
    restart_timeout = "15m"
  }
  ...
  post-processor "amazon-import" {
      aws_polling = {
        delay_seconds = 30
        max_attempts  = 600
    }
  }
}

The documentation for the post-processor only mentions environment variables. In other builds where we are using the amazon-ebs builder we can set the aws_polling configuration on the builder itself, but adding this configuration to the post-processor generates an error message:

Error: Failed preparing post-processor-block "amazon-import" ""
Unsupported argument; An argument named
 "aws_polling" is not expected here. Did you mean to define a block of type "aws_polling"?

Update: it seems I had the configuration wrong, as per the error message which I should have read properly. The aws_polling configuration should be defined as a block, i.e. without the equals sign:

  post-processor "amazon-import" {
      aws_polling {
        delay_seconds = 30
        max_attempts  = 600
    }
  }

Perhaps it would be useful if the documentation for the post-processor mentioned the aws_polling block can be specified on this post-processor too?

from packer-plugin-amazon.