Code Monkey home page Code Monkey logo

awslabs / aws-service-catalog-products Goto Github PK

View Code? Open in Web Editor NEW
156.0 12.0 35.0 2.03 MB

This repository contains a number of CloudFormation templates which can be used independently or as Products with AWS Service Catalog including the Open Source Tools AWS Service Catalog Factory and AWS Service Catalog Puppet. The templates include a number of the foundational AWS Services you may choose to manage Account Compliance including AWS Config, AWS CloudTrail and GuardDuty

License: MIT No Attribution

Python 82.16% JavaScript 1.19% TypeScript 12.35% HTML 3.35% SCSS 0.95%
aws servicecatalog selfservice cloudformation iaas

aws-service-catalog-products's Introduction

AWS Service Catalog Products

This repository contains a number of solutions that can be easily deployed by the Service Catalog Tools (aws-service-catalog-factory and aws-service-catalog-puppet). The solutions are grouped by functional area:

  • foundational - these are the solutions we recommend you use when building a foundation in AWS
  • operations - these are the solutions we recommend to help you with operations
  • reference - these are sample templates useful when you are learning how to write AWS Cloudformation templates
  • unsorted - these are solutions that have been tested but not yet classified or standardised

Getting started

Within the functional area directories you will see a list of solutions:

ls -la foundational

drwxr-xr-x   8 user  group   256 18 Nov 21:49 .
drwxr-xr-x  15 user  group   480 29 Nov 09:55 ..
drwxr-xr-x   7 user  group   224 19 Nov 14:39 amazon-guardduty-multi-account
drwxr-xr-x   6 user  group   192 19 Nov 14:44 aws-securityhub-multi-account
drwxr-xr-x   6 user  group   192 19 Nov 12:58 delete-default-networking

Each solution should be deployable independently and each solution should be deployable with all other solutions in this repository without issue.

Each solution has the same consistent folder structure:

ls -la foundational/amazon-guardduty-multi-account

drwxr-xr-x  7 user  group   224 19 Nov 14:39 .
drwxr-xr-x  8 user  group   256 18 Nov 21:49 ..
-rw-r--r--  1 user  group     0 18 Nov 21:43 README.md
drwxr-xr-x  6 user  group   192 19 Nov 14:07 stacks
drwxr-xr-x  6 user  group   192 19 Nov 14:07 portfolios
drwxr-xr-x  6 user  group   192 19 Nov 14:07 apps
drwxr-xr-x  6 user  group   192 19 Nov 14:07 workspaces
-rw-r--r--  1 user  group  2602 19 Nov 14:39 example-manifest.yaml
  • The README.md explains what the solution does.
  • The stacks directory is the source code for the parts of the solution that should be created as stacks
  • The portfolios directory is the source code for the parts of the solution that should be created as products
  • The apps directory is the source code for the parts of the solution that should be created as CDK apps
  • The workspaces directory is the source code for the parts of the solution that should be created as Terraform workspaces
  • The example-manifest.yaml is a valid YAML file providing an example of how to deploy the solution

Within each of the directorys stacks, portfolios, apps, workspaces you will find a file named amazon-guardduty-multi-account.yaml. Each of the amazon-guardduty-multi-account.yaml files is a valid factory YAML file that can be used to configure your install to build the parts of the solution that are needed.

Importing a solution

To import a solution you should copy each of the YAML files from its stacks, apps, workspaces and portfolios directories into the matching directory in your ServiceCatalogFactory repo. If you are not going to be using AWS CodeCommit for the source directory you will need to change that configuration. Once you are ready commit the changes and run the factory pipeline. The new pipelines will be created and you are now ready to copy over the source code for the different parts.

Deploying a solution

To deploy a solution you should copy the contents of the example-manifest.yaml into your ServiceCatalogPuppet repo into the path manifests/<functional_area>.yaml. If there is a file in that path already you should merge the contents.
Ensure you are happy with the parameters specified and the tags used before committing and pushing the change.

Contributing

If you have a solution you would like to contribute please raise an issue to see verify if a similar solution existing already or is in active development.

Once you are ready to get started please fork the repository and complete the following TODOs:

  1. choose which or create a new functional area your solution should be added to (see above for descriptions)
  2. choose a unique and descriptive name - eg amazon-guardduty-multi-account
  3. create a foundational/amazon-guardduty-multi-account/README.md
  4. read over the programming standards section below
  5. build out your solution in the directories foundational/amazon-guardduty-multi-account/stacks, foundational/amazon-guardduty-multi-account/apps, foundational/amazon-guardduty-multi-account/workspaces, foundational/amazon-guardduty-multi-account/portfolios
  6. create each factory YAML file in each directory - eg. foundational/amazon-guardduty-multi-account/stacks/amazon-guardduty-multi-account.yaml
  7. create an foundational/amazon-guardduty-multi-account/example-manifest.yaml file containing each part of the solution and each of the parameters you have created for each part.
  8. raise a PR

Programming Standards

Please read the following standards and follow them when implementing solutions for this repo

General and Structure

  1. You should not make changes to previously shared parts of your solution - you should create a new version of that part.
  2. For each part of your solution that will be provisioned by the Service Catalog Tools you should use stacks, apps or workspaces - portfolios should be reserved for self service use cases.
  3. When building your solution you should favour mono repos over poly repos - with the exception of when you are using portfolios.
  4. When building solutions where parts need to be provisioned into different accounts try to limit the number of parts needed for each account to 1. This will reduce the complexity for users wishing to deploy your solution.
  5. Each solution should be configured to use AWS CodeCommit as a git source to ensure a consistent experience when users import.

Common parameters and tags

When writing your solution ensure you are using the autogenerated / provided parameters:

  • SCTAccountId - puppet and factory account id
  • SCTManifestAccounts - JSON encoded list of all accounts included in the manifest (including accounts in OUs specified) comprising of account_id and email attributes
  • SCTManifestSpokes - same as SCTManifestAccounts but without including the SCT account.
  • SCTConfigRegions - JSON encoded list of regions specified in the config

You should also use the following tags in your example-manifest.yaml:

  • role:sct - the account containing the Service Catalog Tools
  • role:org_management - the AWS Organizations management account
  • role:securitytooling - the account designated for security tooling
  • role:spoke - spoke accounts

AWS CloudFormation

  1. All AWS CloudFormation parameter and output names should be prefixed with the solution name to avoid parameter clashes.
  2. All AWS CloudFormation templates should pass a CFN Nag check
  3. Each AWS CloudFormation template should have a description
  4. Each AWS CloudFormation template parameter should have a description
  5. Each AWS CloudFormation template output should have a description

AWS IAM Resources

  1. When creating IAM resources you should allow others to configure the path and role name via parameters.
  2. There should be default values for IAM path and role name values
  3. IAM path default values should be the same value across all parts of the solution and should contain the solution category and an indication of the solution name in it.
  4. IAM roles should follow least privilege or users should be able to specify an IAM boundary.

AWS Lambda

  1. AWS Lambda function dependencies should be pinned to specific versions.
  2. AWS Lambda functions should allow the configuration of log level via an environmental variable.

Where is the old content

The previous master branch is still available as the branch archive

License Summary

This sample code is made available under the MIT-0 license. See the LICENSE file.

aws-service-catalog-products's People

Contributors

croccale avatar dependabot[bot] avatar eamonnfaherty avatar hieronymuslex avatar mckayjuk avatar ricmerr avatar smuralee avatar toriancrane avatar tv-17 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

aws-service-catalog-products's Issues

Move-to-OU fails if account already in OU

If the account build fails and is run again, move-to-ou will fail if the account was already moved to the target OU. There should be logic in the code to catch this and just move on, right now it looks like it just hits the catch-all Exception if current_ou != target_ou does not match.

Add SSM support for CodeBuild actions

Current implementation

  • Actions need the code build project name to be hardcoded
actions:
  ping-on-prem-host:
    type: codebuild
    project_name: ping-on-prem-host
    account_id: '0123456789010'
    region: 'eu-west-1'
    parameters:
      HOST_TO_PING:
        ssm:
          name: HOST_TO_PING
          region: eu-west-1

Requested feature

  • Support SSM param for the project_name and account_id
actions:
  ping-on-prem-host: 
    type: codebuild
    project_name: 
      ssm: 
        name: ping-on-prem-host
        region: eu-west-1
    account_id: 
      ssm: 
        name: ping-host-account-id
        region: eu-west-1
    region: eu-west-1
    parameters: 
      HOST_TO_PING: 
        ssm: 
          name: HOST_TO_PING
          region: eu-west-1

move-to-ou being invoked with null for account_id

On a new account creation run using AVM, the move-to-ou function fails on the first run with a null value passed:

Sample log snippet:

START RequestId: 12ca5541-e4de-46a0-b495-f2614f2e9412 Version: $LATEST
--
[INFO]	2020-12-08T21:40:48.967Z	12ca5541-e4de-46a0-b495-f2614f2e9412	Create
[INFO]	2020-12-08T21:40:49.253Z	12ca5541-e4de-46a0-b495-f2614f2e9412	Found credentials in environment variables.
[INFO]	2020-12-08T21:40:50.513Z	12ca5541-e4de-46a0-b495-f2614f2e9412	list_parents: {'ChildId': 'null'}
[INFO]	2020-12-08T21:40:50.514Z	12ca5541-e4de-46a0-b495-f2614f2e9412	searching, NextToken: FirstPage
[ERROR]	2020-12-08T21:40:50.713Z	12ca5541-e4de-46a0-b495-f2614f2e9412	An error occurred (InvalidInputException) when calling the ListParents operation: You provided a value that does not match the required pattern.
[INFO]	2020-12-08T21:40:50.982Z	12ca5541-e4de-46a0-b495-f2614f2e9412	Status message: OK 200
END RequestId: 12ca5541-e4de-46a0-b495-f2614f2e9412
REPORT RequestId: 12ca5541-e4de-46a0-b495-f2614f2e9412	Duration: 2026.49 ms	Billed Duration: 2027 ms	Memory Size: 128 MB	Max Memory Used: 76 MB	Init Duration: 567.75 ms

If the AVM product is run a second time it works w/o issue.

CodeBuild for cdk deploy incorrect system parameter passing

In the CloudFormation created when a product is installed the yaml contains the following parameter:
'BootstrapVersion': 'Default': '/cdk-bootstrap/hnb659fds/version' 'Description': 'Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. [cdk:skip]' 'Type': 'AWS::SSM::Parameter::Value<String>'
This automatically resolves the parameter store value of -/cdk-bootstrap/hnb659fds/version-

When the build tries to run with the cdk deploy command it passes the actual value in the variable CDK_DEPLOY_PARAMETER_ARGS rather than the parameter store name.
e.g.
Currently passes:

--parameters HelloCdkStack:BootstrapVersion=12

Should pass: -

-parameters HelloCdkStack:BootstrapVersion=/cdk-bootstrap/hnb659fds/version

This cause the install to fall over with a validation error:

Error [ValidationError]: Unable to fetch parameters [12] from parameter store for this account

Only workarounds identified are to hard code the parameter name into the lamdba servicecatalog-tools--StartCDKDeploy or to set up a parameter in system manager manually called 12 with a value of 12...

mulit-account-aws-config

According to the docs the following products are missing:

  • aws-config-sns-hub
  • aws-config-sns-spoke

Service catalog dashboard permission issues

Below error is seen while provisioning the static website product for the dashboard

ERROR: [pid 419] Worker Worker(salt=278550791, workers=10, host=d105511e43cc, username=root, pid=354) failed CreateAssociationsInPythonForPortfolioTask(manifest_file_path=manifest-expanded.yaml, puppet_account_id=564683158468, account_id=123456789012, region=eu-west-1, portfolio=reinvent-cloud-engineering-governance, portfolio_id=port-tix4qsnkpbde2) Traceback (most recent call last): File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/luigi/worker.py", line 191, in run new_deps = self._run_get_new_deps() File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/luigi/worker.py", line 133, in _run_get_new_deps task_gen = self.task.run() File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/servicecatalog_puppet/workflow/portfoliomanagement.py", line 1501, in run PrincipalType="IAM", File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call return self._make_api_call(operation_name, kwargs) File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/botocore/client.py", line 676, in _make_api_call raise error_class(parsed_response, operation_name) 

error - botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the AssociatePrincipalWithPortfolio operation: User: arn:aws:sts::123456789012:assumed-role/SC-123456789012-pp-dly634ma-DashboardCodeBuildRole-151IVMZ0DV3WG/AWSCodeBuild-12e35240-9c81-459d-9f1c-b06f8b6f6d32 is not authorized to perform: servicecatalog:AssociatePrincipalWithPortfolio on resource: arn:aws:catalog:eu-west-1:564683158468:portfolio/port-tix4qsnkpbde2

account-vending

In the Account Vending Machine bootstrapping product, there needs to exist a wait time (e.g. 2mins) to allow for the account creation to complete.

Suggested fix would be to add time.sleep(120) to line 40 for example of the following file:
aws-service-catalog-products/account-vending/account-bootstrap-shared/v1/src/handler.py

Standard Expression failure in servicecatalog-tools--StartCDKDeploy code

When trying to run the install of a cdk package through service catalog using the supplied tools there are issues with the integration with crhelper.

Firstly, as crhelper is not on AWS as a standard layer the code does not work out of the box. It would be helpful to either provide information about the creation of a lambda layer (with a variable of that lambda layer to pass in, and preferably instructions for IaC rather than console) or to provide the correct version of crhelper that should be used as part of the package to be deployed.

The main issue, however, is the integration between the lambda and crhelper.
In crhelper code file resource_helper.py for version 2.0.10 line 271 a variable ScheduledExpression is set using the value of polling_interval from the CfnResource class definition.
At the top of the lambda servicecatalog-tools--StartCDKDeploy helper is set to be the CfnResource but with a polling_interval=1
This results in the following:
ScheduleExpression='rate(1 minutes)'
Notice the pleural of minutes is still there. This causes an issue as follows:
image

I have not come across a version of crhelper which deals with this more effectively so the easiest work around for this is to set the polling_interval to be 2 instead at the top of the Lambda.

account-creation-shared declares account_id early

https://github.com/awslabs/aws-service-catalog-products/blob/master/governance-at-scale-account-factory/account-creation-shared/v3/src/handler.py

The account_id var is declared with the initial response to CreateAccount, which doesn't contain an AccountId field. As a result, a null value is passed on to the downstream bootstrapping, which then fails.
The account_id declaration for new accounts should be moved to after the DescribeCreateAccountStatus call.

ResourceNotFound Exception in Puppet run

Intermittently receiving this error during a Puppet run:

Traceback (most recent call last):
  File "/root/.pyenv/versions/3.8.3/lib/python3.8/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/root/.pyenv/versions/3.8.3/lib/python3.8/site-packages/luigi/worker.py", line 141, in _run_get_new_deps
    task_gen = self.task.run()
  File "/root/.pyenv/versions/3.8.3/lib/python3.8/site-packages/servicecatalog_puppet/workflow/provisioning.py", line 70, in run
    path_id = aws.get_path_for_product(
  File "/root/.pyenv/versions/3.8.3/lib/python3.8/site-packages/servicecatalog_puppet/aws.py", line 406, in get_path_for_product
    response = service_catalog.list_launch_paths(ProductId=product_id)
  File "/root/.pyenv/versions/3.8.3/lib/python3.8/site-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/root/.pyenv/versions/3.8.3/lib/python3.8/site-packages/botocore/client.py", line 635, in _make_api_call
    raise error_class(parsed_response, operation_name)

botocore.errorfactory.ResourceNotFoundException: An error occurred (ResourceNotFoundException) when calling the ListLaunchPaths operation: No launch paths found for resource: prod-xxx

(xxx being product id)

Running the puppet pipeline again right after usually resolves the issue. This error occurred during the single-account-run-with-callback CodeBuild job, as part of the AVM flow for a new account creation. The product it is erring on was already deployed, and successfully run against new accounts with a full Puppet CodePipeline run.

account-waiter has incorrect syntax for iam role

governance-at-scale-account-factory/account-waiter/v3/product.template.yaml

AccountWaiterCustomResourceRole:
    Type: AWS::IAM::Role
    Properties:
      Path: !Ref GovernanceAtScaleAccountFactoryIAMRolePath
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: ServiceCatalogActions
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - sts:AssumeRole
                Resource: !Ref GovernanceAtScaleAccountFactoryAccountCreationSharedOrgRoleArn
              - Effect: Allow
                Action:
                  - codebuild: '*'
                Resource: '*'
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              AWS: !Sub "arn:aws:iam::${AWS::AccountId}:root"
            Action:
              - "sts:AssumeRole"
          - Effect: "Allow"
            Principal:
              Service:
                - "lambda.amazonaws.com"
            Action:
              - "sts:AssumeRole"

codebuild: '*' should be - 'codebuild:*'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.