aws-samples / aws-iam-access-key-auto-rotation Goto Github PK

We get a lot of notifications from this, roughly one per day leading up to the rotation itself which could be like 15 emails for one rotation, is there a way outside of modifying the lambda functions themselves to reduce the notifications? We have no problems doing that, just wondered whether this was something you already planned to solve.

I think having the notification schedule be a bit more configurable would really benefit the users of the library.

I cloned the repo and in my Lambda subdirectory there are only three zip files. Yet, step 3.0 of the "ASA IAM Key Rotation Runbook(v3).pdf" it shows a picture that has these zips and directories with their names. Is there a step missing where I should have extracted these zip files? It also has a sub-heading of "Project files included in the zip:" but this stuff was cloned from github.

Later in the Runbook it shows a screen capture of copying things into a lambdacoderepo1221155/asa/asa-iam-rotation/
but then in step 4.2 Step 3 it shows a picture of ****-demo-bucket-iam-key-rotation. What is the form of this CloudFormation S3 Bucket Name? Using your example should this something like s3://lambdacoderepo1221155?

For 4.2 Step 4 the Runbook says "For Permissions, select 'Service-managed permissions'..." but shows a screen shot that selects 'Self service permissions' with an IAM role name of AWSCloudFormationStackSetAdministrationRole and an IAM execution role name.

ResourceLogicalId:RotationAccessKeyRotateLambdaFunction, ResourceType:AWS::Lambda::Function, ResourceStatusReason:1 validation error detected: Value 'arn:aws:s3:::testaccessrotation' at 'code.s3Bucket' failed to satisfy constraint: Member must satisfy regular expression pattern: ^[0-9A-Za-z.-_]*(?<!.)$ (Service: AWSLambdaInternal; Status Code: 400; Error Code: ValidationException; Request ID: cf2f8d55-56b5-4414-8060-179908e15edf; Proxy: null)

Issue 1: This function seems to update secrets manager only in plaintext format not in key pair format.
Issue 2: if there is already an entry for user in key value pair in secrets manager , (let say when user is created via CF and updated to SM). those keys are not updated at all.

in both scenarios keys gets rotated and i get notification., but there is not key update on secrets manager.
i can confirm there is gap in the code and is partially running to even test for test scenarios. gap in this solution is implementation and how console or aws handles secrets in json format.

The runbook instructions seem to be rather inaccurate and out of order. Here are a few issues I've noticed so far

Deployment steps begins with uploading the files to S3 and deploying the assumed roles yaml template as a stack set. This will immediately fail as the execution role does not yet exist:

ResourceLogicalId:ASAIAMAssumedRole, ResourceType:AWS::IAM::Role, ResourceStatusReason:Invalid principal in policy: "AWS":"arn:aws:iam::999999999999:role/asa-iam-key-rotation-lambda-execution-role" (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxx; Proxy: null).

The role is created under the 'ASA-iam-key-auto-rotation-and-notifier-solution' stack - which according to these instructions is the last thing deployed.

4.2 Step 3: "You will need to enter the ‘Primary AWS Account ID’ and ‘AWS Organization ID’."

There is no parameter for providing an org id. The included screenshot shows only the account id is needed.

4.2 Step 4: Select ‘Service-managed permissions’

This option appears on the first page of the create stackset wizard, not the page after providing the parameters. The included screenshot has 'Self service permissions' selected - which is the correct option?

4.3 Step 3: Deploy the List Account Role in the Central/Management Account.

Says a stack needs to be deployed but does not say which. Immediately jumps to filling in the parameters.

I'm sure there are more issues, however the quality of this documentation for a solution advertised by AWS is rather poor and puts me off using it altogether.

I have a use case that I want to to the rotation for a specific IAM user.
Is it possible to configure it?

I've deployed the solution however the secret manager is not created in any of the accounts. I validated 2 of the regions where the solution was deployed.

For testing purpose, I have set up the Rotation period to 1 (one) - leaving everything else as default (inactive period, deletion period, etc)

For my surprise it deleted the Access keys of User instead of Rotating it. I tried again and same thing happened. Instead of rotating it deleted the user's key when rotation period was set to 1.

Hi when I try to manually test the ASA-IAM-Access-Key-Rotation-Function Lambda Function it appears that I keep receiving:

Error: An error occurred (AccessDeniedException) when calling the ListAccounts operation: You don't have permissions to access this resource.

I can see that the policies attached to the resource allows for the ListAccounts operation, but for some reason I keep receiving the above error.

Here are some screenshots for more context:

Also thank you for the updated version of this tool 🙏

As mentioned in #21, the documentation has a lot of different points from what is exactly been implanted. Would be fine if someone of the team can take a look again to it.

I cant deploy the first step, I keep receving messages when I try to create the stacks at my organizations accounts:

ResourceLogicalId:ASAIAMAssumedRole, ResourceType:AWS::IAM::Role, ResourceStatusReason:Invalid principal in policy: "AWS":"arn:aws:iam::978768682946:role/asa-iam-key-rotation-lambda-execution-role" (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: 0dec4956-d5f9-4839-93ce-7d1da67cdd04; Proxy: null).

At the documentation I can't find any other step before this.

As per the title, I want to look into implementing this. I would like to apply it to some test accounts for POC, then Dev, then STG then Prod over a period of time. I don't want to have to go to 150+ accounts and create the "IAMKeyRotationExemptionGroup" and move everyone in there, as I wont get approval before the demo is complete and documented.

I am wondering if there is a way to set a tag on the accounts, or just supply a list of account numbers to the scan so i can do this in a controlled manner?
Alternatively is it possible to supply a specific OU in the CTOrganisation?

Hi, I have followed the setup but encountered this error, can you help me what's wrong? I am not very familiar with lambda.

[ERROR] ParamValidationError: Parameter validation failed: Invalid type for parameter AccessKeyId, value: {'UserName': 'iam-user', 'AccessKeyId': 'XXXXXXXXXXXXXXXXX', 'Status': 'Inactive', 'CreateDate': datetime.datetime(2020, 9, 29, 11, 29, 30, tzinfo=tzlocal())}, type: <class 'dict'>, valid types: <class 'str'> Traceback (most recent call last): File "/var/task/main.py", line 107, in lambda_handler process_user(user, response) File "/var/task/main.py", line 223, in process_user iam.delete_access_key(UserName=user_name, File "/var/runtime/botocore/client.py", line 357, in _api_call return self._make_api_call(operation_name, kwargs) File "/var/runtime/botocore/client.py", line 648, in _make_api_call request_dict = self._convert_to_request_dict( File "/var/runtime/botocore/client.py", line 696, in _convert_to_request_dict request_dict = self._serializer.serialize_to_request( File "/var/runtime/botocore/validate.py", line 297, in serialize_to_request raise ParamValidationError(report=report.generate_report())

Error while getting file contents for iam-auto-key-rotation-enforcement.html - An error occurred (AccessDenied) when calling the GetObject operation: Access Denied

Got everything working but the above. Can't figure out why I am getting access denied error...

I'm testing it in my personnel account where I have created my Organization by inviting my existing account.
I have followed every single step as per the Document Version 2.0 but its keep failing at Step number 6 in 4.2 and Step number 6 in 4.3.

Errors: -

ResourceLogicalId:NotifierLambdaFunction, ResourceType:AWS::Lambda::Function, ResourceStatusReason:Resource handler returned message: "Your access has been denied by S3, please make sure your request credentials have permission to GetObject for keyrotationpocjyo/asa/asa-iam-rotation/Lambda/notifier.zip. S3 Error Code: AccessDenied. S3 Error Message: Access Denied (Service: Lambda, Status Code: 403, Request ID: 4f4fe2e9-09c9-4ac5-8e62-385d59b0d073)" (RequestToken: 4aa078bf-0364-44e0-8cdd-c6de2ae3b422, HandlerErrorCode: AccessDenied).

ResourceLogicalId:ASAIAMExemptionsGroup, ResourceType:AWS::IAM::Group, ResourceStatusReason:IAMKeyRotationExemptionGroup already exists.

I just followed the document no additional steps I have done pre/post of the guidelines in the document, can you please suggest, what action has to be take to fix this.

I understood from the Error Status reason that I need to set some permissions but wondering why they are not defined in the document? Can you please suggest to fix this issue.

In access_key_auto_rotation\account_scan.py one adds resource_email

            # Update actions with resource owner email from tag
            if config.resourceOwnerTag is not '':
..
                    for action in user_actions:
                        action.update({"resource_email": resource_owner_email})

but in access_key_auto_rotation\main.py one is looking for resource_owner

        # Extract subsets of actions for resource owners
        resource_owners = {action.get("resource_owner") for action in action_queue}

resulting no resource_owners being found ..

I deploy the cloudformatoin template in singpaore region but secret keys are created in us-east-1.
How do I fix this.

Thank you.

Hi, thanks for your input. Tried to create the stack. It works fine to scan and create the IAM access key after 90 days. However, no SNS message was received. Cross-checked the Cloudwatch Event rules history in RotationCloudWatchEventSNS, it never run.

Any idea?
Any coding is required to update in "IAM-Access-Key-Rotation-LambdaFunctionName"

Thanks.

Issue 1: This function seems to update secrets manager only in plaintext format not in key pair format.
Issue 2: if there is already an entry for user in key value pair in secrets manager , (let say when user is created via CF and updated to SM). those keys are not updated at all.

in both scenarios keys gets rotated and i get notification., but there is not key update on secrets manager.
i can confirm there is gap in the code and is partially running to even test for test scenarios. gap in this solution is implementation and how console or aws handles secrets in json format.

This has been very helpful and worked well on my test with a single account.

When Deploying as a stackset for an Org, the current template tries to create a bucket for each of the accounts based on the CSVBucketName parameter. The first stack that gets created works fine but subsequent accounts fail as the bucket name is not unique now that it exists in the first account.

I was able to work around this by replacing the call to !RefCSVBucketName to the following
!Join [ "-", [!Ref CSVBucketName, !Ref AWS::AccountId]]

Additionally, a bucket policy needs to be added to the bucket used in the S3BucketName parameter to allow the accounts in the org to GetObject and List Object

I now need to sort out the SES outbound from all the accounts for the notification. Any way that could come from a single account?

Running into the following error after deploying cloudformation for password rotation.

6858314d-cee5-4cf8-abfd-392603586515 Check that AccountID: [AccountID] has the correct IAM Role deployed to it via the CloudFormation Stack Template. Raw Error: An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::AccountID:assumed-role/asa-iam-key-rotation-account-inventory-lambda-execution-role/ASA-Account-Inventory is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::AccountID:role/asa-iam-key-rotation-list-accounts-role

policy has assume role for specified role in error.

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "lambda:InvokeFunction",
"Resource": "*"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::AccountID:role/asa-iam-key-rotation-list-accounts-role"
}
]
}

From force_rotation_handler.py in access_key_auto_rotation.zip

def check_forced_rotate_flag(event, noUsers, log):
    # Initialize Values
    force_rotate = None
    force_rotate_user_name = None

    log.info('Checking if ForceRotate flag exists.')

    # Check if the message sent to the Lambda contained the value 'ForceRotate'
    # Note: This currently only supports one username at a time for testing
    if "ForceRotate" in event and not noUsers:
        force_rotate_user_name = event['ForceRotate']
        force_rotate = True
        log.info(f'ForceRotate flag exists for [{force_rotate_user_name}].')
    elif "ForceRotate" not in event and not noUsers:                                        # Same expression as below elif
        force_rotate = False
        log.info(
            'ForceRotate flag does not exist and '
            'there are users in this account.')
    elif "ForceRotate" not in event and not noUsers:                                       # Same expression as above elif
        log.info(
            f'ForceRotate flag exists for [{force_rotate_user_name}]'
            f' but there are no users in this account.')
        force_rotate = True
    else:
        log.error('Undetected type. Listing noUsers(boolean) and users(array)')
        force_rotate = False

    return force_rotate, force_rotate_user_name

From the log messages, it is my assumption the second elif should instead be:

 elif "ForceRotate" in event and not noUsers:

I've found the incorrect order of cloudformation deployment which is leading to users to get deploy rolled back due to it.

The correct order of cloudformation templates is:

1- [ASA-iam-key-auto-rotation-and-notifier-solution.yaml]
2- [ASA-iam-key-auto-rotation-iam-assumed-roles.yaml]
3- [ASA-iam-key-auto-rotation-list-accounts-role.yaml]
4- [ASA-iam-key-auto-rotation-vpc-endpoints.yaml] - This is optional.

You can also find the correct sequence in the AWS page: https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/automatically-rotate-iam-user-access-keys-at-scale-with-aws-organizations-and-aws-secrets-manager.html

running ASA-iam-key-auto-rotation-list-accounts-role.yaml

Invalid principal in policy: "AWS":"arn:aws:iam::899720958166:role/asa-iam-key-rotation-account-inventory-lambda-execution-role" (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: 5a36253e-7d8f-47ee-8457-70974ed706c6; Proxy: null)

Version 2 Release -

config.py in access_key_auto_rotation refers to environmental variables InstallationGracePeriod and PendingActionWarnPeriod.

However, I'm having trouble finding those as part of the cloudformation installation for that function.

I have InactivePeriod and InactiveBuffer as environmental variables that don't seem to be referenced.

Can you point out what I've misunderstood?

Thanks.