Code Monkey home page Code Monkey logo

aws-codecommit-serverless-backup's Introduction

Introduction

AWS CodeCommit is a fully-managed source control service that makes it easy for companies to host secure and highly scalable private Git repositories. CodeCommit eliminates the need to operate your own source control system or worry about scaling its infrastructure. You can use CodeCommit to securely store anything from source code to binaries, and it works seamlessly with your existing Git tools.

You typically don't need to worry about backing up your CodeCommit repositories as CodeCommit's architecture is highly scalable, redundant, and durable. However, there are situations where backups might be helpful. For instance, if one accidentally deletes the CloudFormation stack that created the CodeCommit repository, the entire repository and its contents are also deleted for good. Oops.

As per AWS documentation: "Deleting an AWS CodeCommit repository is a destructive one-way operation that cannot be undone. To restore a deleted repository, you will need to create the repository again and use either a backup or a local copy from a full clone to upload the data".

So, having a backup handy is not a bad idea - Better safe than sorry!

The Solution

This project offers a serverless CodeCommit backup solution (who wants to manage servers these days?) that uses an Amazon CloudWatch event rule as a trigger (eg, trigger the backup every day at 2am UTC). (See the figure below for details). The CloudWatch event targets an AWS Lambda function that simply triggers an AWS CodeBuild container that generates a backup of all AWS CodeCommit repositories within a particular AWS account and region. The backup consists of .tar.gz files named after the repository's name and using a timestamp (eg, Repo1_2017_10_01_02_00). The backups are stored in a designated S3 bucket (eg, backup-bucket/Repo1, backup-bucket/Repo2, etc). One can use S3 lifecycle events to automatically move old backups into Amazon Glacier (cold storage) or alternatively specify an expiration policy for backup files in S3 to have them deleted after a certain period of time. Also as a security best practice, the S3 bucket storing the backups should enable default encryption.

approach-overview

Deploying the Solution

  • Clone this project in your local workstation (only tested on Mac OS)
  • Make sure you have the latest version of the AWS Command Line Interface installed in your local box
  • Make sure the user in your AWS profile have permissions to create IAM roles, CloudWatch Event rules, CodeBuild projects and Lambda functions at a minimum (check your profiles under ~/.aws/credentials)
  • Open script ./deploy.sh and update these parameters as desired: AWS profile, S3 buckets, and backup schedule. The S3 buckets will not be created. They must exist already. (See parameters below.)
aws_profile="default"                  # default AWS profile (or choose another profile)
backup_schedule="cron(0 2 * * ? *)"    # backups scheduled for 2am UTC, everyday
scripts_s3_bucket="[S3-BUCKET-FOR-BACKUP-SCRIPTS]s" # bucket must exist in the SAME region the deployment is taking place
backups_s3_bucket="[S3-BUCKET-FOR-BACKUPS" # bucket must exist and have no policy that disallows PutObject from CodeBuild
stack_name="codecommit-backups"        # CloudFormation stack name for the solution
  • By default, all CodeCommit repositories within the AWS region where the solution was deployed to will be backed up everyday at 2am UTC (cron(0 2 * * ? *)) into the S3 bucket specified
  • Run script ./deploy.sh to deploy the solution into your AWS account
  chmod +x ./deploy.sh
  ./deploy.sh

Catch failed runs

  • Create an AWS SNS email topic
  • Create a CloudWatch Event like this one:
{
  "source": [
    "aws.codebuild"
  ],
  "detail-type": [
    "CodeBuild Build State Change"
  ],
  "detail": {
    "build-status": [
      "FAILED"
    ],
    "project-name": [
      "CodeCommitBackup"
    ]
  }
}
  • Assign the SNS topic as a target for this event

License Summary

This sample code is made available under the MIT-0 license. See the LICENSE file.

aws-codecommit-serverless-backup's People

Contributors

chris-redekop avatar hyandell avatar jazzl0ver avatar jpeddicord avatar marcilio avatar markatwood avatar meckhardt avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

aws-codecommit-serverless-backup's Issues

Getting S3 access error

When I run this example it seems to be failing for me.

I've created 2 buckets with I think the proper permissions.
When I run this code the package statement creates a new 'file' (57bd91439d8450ae073296f4f12d3ca2) in my bucket the same size of the codecommit_backups.zip file...
And the generated template has a CodeURI added to it for the lambda function:
CodeUri: s3://mybackupbucketscripts/57bd91439d8450ae073296f4f12d3ca2
but the cloudformation deploy step fails and the stack is saying:

Your access has been denied by S3, please make sure your request credentials have permission to GetObject for mybackupbucketscripts/57bd91439d8450ae073296f4f12d3ca2

When I goto S3 in teh console this file has no permissions on it that I can seem to access. Is there a setup step I'm missing?

bug in aws_profile under deploy.sh

If you have multiple AWS credentials, make sure you set your env as [default] under ~/.aws/credential. Otherwise, using the aws_profile under the deploy.sh will cause error below:

Unable to upload artifact None referenced by CodeUri parameter of CodeCommitBackupLambda resource.

An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

Codebuild fails with tar command

After git clone, the tar command fails, anyone encounter this issue?

`
02:01:10
Cloning into bare repository 'ep-water-admin.git'...

02:01:30
remote: remote: Counting objects: 0 remote: Counting objects: 0 remote: Counting objects: 0 remote: Counting objects: 0 remote: Counting objects: 0 remote: Counting objects: 0 remote: Counting objects: 0 remote: Counting objects: 0 remote: Counting objects: 0 remote: Counting objects: 0 remote: Counting objects: 0 remote: Counting objects: 0 remote: Counting objects: 0 remote: Counting objects: 2

02:02:10
Receiving objects: 0% (1/2332) Receiving objects: 1% (24/2332) Receiving objects: 2% (47/2332) Receiving objects: 3% (70/2332) Receiving objects: 4% (94/2332) Receiving objects: 5% (117/2332) Receiving objects: 6% (140/2332) Receiving objects: 7% (164/2332) Receiving objects: 8% (187/2332) Receiving objects: 9% (210/2332) Receiving objects: 10% (2

02:02:10
Receiving objects: 100% (2332/2332) Receiving objects: 100% (2332/2332), 636.58 KiB | 2.29 MiB/s, done.

02:02:10
Resolving deltas: 0% (0/1395) Resolving deltas: 1% (16/1395) Resolving deltas: 4% (63/1395) Resolving deltas: 6% (84/1395) Resolving deltas: 8% (123/1395) Resolving deltas: 10% (144/1395) Resolving deltas: 11% (163/1395) Resolving deltas: 12% (171/1395) Resolving deltas: 13% (182/1395) Resolving deltas: 15% (210/1395) Resolving deltas: 16% (228/1395)

02:02:10
++ date -u +%Y_%m_%d_%H_%M

02:02:10

  • dt=2018_08_28_02_02

    02:02:10
  • zipfile=ep-water-admin_backup_2018_08_28_02_02_UTC.tar.gz

    02:02:10
  • echo 'Compressing repository: ep-water-admin into file: ep-water-admin_backup_2018_08_28_02_02_UTC.tar.gz and uploading to S3 bucket: rc-codecommit-backup-repos-ap-northeast-1/ep-water-admin'

    02:02:10
    Compressing repository: ep-water-admin into file: ep-water-admin_backup_2018_08_28_02_02_UTC.tar.gz and uploading to S3 bucket: rc-codecommit-backup-repos-ap-northeast-1/ep-water-admin

    02:02:10
  • tar -zcvf ep-water-admin_backup_2018_08_28_02_02_UTC.tar.gz ep-water-admin/

    02:02:10
    tar: ep-water-admin: Cannot stat: No such file or directory

    02:02:10
    tar: Exiting with failure status due to previous errors`

cloudbuild fail with No permission error to /bin/bash: ./backup_repo.sh: Permission denied

Hi,

I tried to test code on AWS. Stack get created completely but when I tried to test it with code build it gives an error

[Container] 2022/09/10 09:28:20 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: ./backup_codecommit.sh. Reason: exit status 39

In lambda function I changed python version 3.6 to 3.9 only. in deploy.sh file.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.