It would be nice if we could run the integration tests as part of the build as well. These tests would need to:

1. Start OPA pointed at the policy directory.
2. Run ./test-templates.py (which we'll need to modify to exit with an error code on test failures).
3. Stop OPA (optional I guess for GH actions, but nice to be able to run this locally as well).

Does the AWS Secrets Manager bearer token value need parsing? Your code:

https://github.com/StyraInc/opa-aws-cloudformation-hook/blob/main/hooks/src/styra_opa_hook/handlers.py#L28-L44

Returns the SecretString from SecretsManager: {"opa_auth_token": "<VALUE>"}.

Are you expecting that entire string {"opa_auth_token": "<VALUE>"} to be configured in the OPA sever token auth? Or do we need to parse the actual value out with something like:

`if 'SecretString' in resp:
    return list(json.loads(resp['SecretString']).values())[0]`

While we can add these manually to styra-opa-hook.json during development as we add more and more resource type example policies, we'll want to list all resource types available for AWS Cloudformation before we publish this hook to the marketplace. The reason for this is that we'll want to keep the hook as generic as possible, and have the user determine whether something is allowed or not in policy rather than configuration.

Ideally, I think we'll have a script run nightly (or whatever interval seems appropriate) to fetch the resource definitions from the AWS API and make sure we're up to date. If we aren't we should create a PR and later on a release from that (this part could be done manually).

I think this is due to a more recent version of Python now coming with the runners, but I'm not sure. We should look into it.

Example: https://github.com/StyraInc/opa-aws-cloudformation-hook/actions/runs/8170632332

The resource schema definition for the AWS resources clearly state that some properties are boolean. Yet, when sent to OPA, they are presented as strings ("true"/"false"). We should see why this happens and fix it before we publish anything.

Users of this hook likely will want to provide their own policies rather than the examples we provide. I think we should move these to a new examples directory where we can have one subdirectory for the templates (as suggested by @tsandall elsewhere), and another for policy.

Remaining in the policy directory should only be main.rego and authz.rego, plus a subdirectory for the tests / helpers.

While YAML files definitely are easier on the eye, JSON is also a supported format, and we should allow that. While this makes no difference to the policy, we should modify the template test runner to allow for this, as that might be what some users provide.

Having the user install the hook themselves works pretty well, but it would be nice if we also offered this hook directly in the AWS Marketplace. This would also help with discoverability.

Right now when the hook fails all we see is: The following hook(s) failed: [Styra::opa::Hook] is it possible to populate the error messages generated in DAS back to CloudWatch and CloudFormation?

The script currently reads the templates directory for all templates, and submits them to OPA for validation one by one.

It would be a great improvement if we instead allowed the user to point the tool at either a single template, or a directory. This would allow them to use the same tool for their own policy authoring, where they'd run OPA locally, and submitted templates converted to JSON to OPA, and have the script return the output. Suggested steps, but these are very much just some ideas:

• Rename to validate.py (or whatever is appropriate)
• Allow argument to point out single template, or directory, which will be submitted to OPA (localhost:8181 hardcoding is fine).
• By default, we shouldn't "test" the returned response but rather just print it to the console, nicely formatted.
• When provided a --test flag or similar, we do what we currently do and print FAIL, SUCCESS, and so on.
• Add some docs to explain what the tool does and how to use it.

After setting up the CloudFormation hook and creating a stack with an S3 bucket with public access, the following was logged to CloudWatch for the hook:

Exception caught sequence item 0: expected str instance, dict found
Traceback (most recent call last):
  File "/var/task/cloudformation_cli_python_lib/hook.py", line 273, in __call__
    raise error
  File "/var/task/cloudformation_cli_python_lib/hook.py", line 262, in __call__
    caller_sess, request, invocation_point, callback, type_configuration
  File "/var/task/cloudformation_cli_python_lib/hook.py", line 100, in _invoke_handler
    return handler(session, request, callback_context, type_configuration)
  File "/var/task/styra_opa_hook/handlers.py", line 127, in pre_handler
    return opa_query(request, session, type_configuration, action)
  File "/var/task/styra_opa_hook/handlers.py", line 97, in opa_query
    message = " | ".join(body["violations"])
TypeError: sequence item 0: expected str instance, dict found

For reference, the OPA agent returned the following:

{"allow":false,"violations":[{"allowed":false,"message":"public access not blocked for bucket testing"}]}

For users who do not have Docker Desktop installed locally due to licensing issues, the cfn submit command cannot be run with the default Docker flow. Disabling the Docker option in the hook configuration and building the zip file on a non-Linux system results in Lambda function errors. The cfn cli does not currently officially support Docker alternatives.

I recommend running the cfn submit --dry-run command on an Ubuntu GitHub runner with Docker installed to generate a valid hook zip file.

This issue is connected with https://stackoverflow.com/questions/72526742/cloudformation-hooks-do-not-push-logs-to-cloudwatch-logs

For the readme, we should have a diagram that shows how data flows from AWS to OPA and back.

While ngrok works great for local testing, we'll not want to depend on a proprietary third-party service for the OPA docs. It would be good if we, as part of the docs, set up OPA either running as a container in something like ECS, or as a Lambda. We'll also want to run this so that it uses the management APIs like how we do it in the other tutorials.