Code Monkey home page Code Monkey logo

aws-dev-hour-backend's Introduction

AWS Dev Hour - Series 1 Demo Application (Backend)

Type: Demo

AWS Dev Hour - On Twitch

Do you have the skills it takes to build modern applications that are distributed and designed for scale and agility? If you’re interested in learning to build cloud native applications and architecture practices, join us for AWS Dev Hour: Building Modern Apps, a weekly Twitch show presented by AWS Training and Certification. Built by developers for developers, the series offers a hands-on approach. Over the course of 8 episodes, AWS expert hosts Ben Newton and May Kyaw will take you through the end-to-end build of a serverless application in the AWS cloud. You’ll have the chance to learn by doing, following along with the hosts and developing a cloud-native application using the AWS free tier. You’ll learn best practices for modern applications and better understand how AWS cloud-native applications differ from on-premises. Throughout the series, you’ll receive code, white papers, links to documentation, and other resources to help you progress.

Architecture

architecture-screenshot

Episodes

During each episode, we will be progressively building this full-stack application together. Please see the following URL for schedule and episode details:

AWS Dev Hour Schedule

Git branch

Episode 1: CDK

Episode 2: AWS Lambda & Amazon DynamoDB

Episode 3: Amazon API Gateway

Episode 4: AWS IAM & Amazon Cognito

Episode 5: Amazon S3

Episode 6: Amazon SQS

Episode 7: Deployment Pipeline

Services used during the series:

  • Amazon Cognito
  • Amazon S3
  • Amazon Simple Queue Service
  • AWS Lambda
  • Amazon DynamoDB
  • Amazon Rekognition
  • AWS Cloud Development Kit
  • Amazon API Gateway
  • AWS CodeBuild
  • AWS CodePipeline

Useful commands

  • npm install install packages
  • cdk synth emits the synthesized CloudFormation template
  • cdk deploy deploy this stack
  • cdk diff compare deployed stack with current state

The cdk.json file tells the CDK Toolkit how to execute your app.

Prerequisites

All CDK developers need to install Node.js 10.3.0 or later, even those working in languages other than TypeScript or JavaScript. The AWS CDK Toolkit (cdk command-line tool) and the AWS Construct Library run on Node.js. The bindings for other supported languages use this back end and tool set. We suggest the latest LTS version.

aws configure
npm -g install typescript
npm install -g aws-cdk

If you have not yet done so, you will also need to bootstrap your account:

cdk bootstrap aws://ACCOUNT-NUMBER-1/REGION-1

for example:

cdk bootstrap aws://123456789012/us-east-1

For further information, please see:

https://docs.aws.amazon.com/cdk/latest/guide/getting_started.html

https://docs.aws.amazon.com/cdk/latest/guide/bootstrapping.html

AWS Lambda - Creating assets for your AWS Lambda Layer

Our AWS Lambda function uses the Pillow library for the generation of thumbnail images. This library needs to be added into our project so that we can allow the CDK to package it and create an AWS Lambda Layer for us. To do this, you can use the following steps. (Please note, creating these resources in your AWS account could incur costs, although we have tried to select free-tier eligble resources here).

  1. Launch an Amazon EC2 Instance (t2-micro) using the Amazon Linux 2 AMI
  2. SSH into your instance and run the following commands:
sudo yum install -y python3-pip python3 python3-setuptools
python3 -m venv my_app/env
source ~/my_app/env/bin/activate
cd my_app/env
pip3 install pillow
cd /home/ec2-user/my_app/env/lib/python3.7/site-packages
mkdir python && cp -R ./PIL ./python && cp -R ./Pillow-8.1.0.dist-info ./python && cp -R ./Pillow.libs ./python && zip -r pillow.zip ./python
  1. Copy the resulting archive 'pillow.zip' to your development environment (we used an Amazon S3 bucket for this)
  2. Extract the archive into the 'reklayer' folder in your project directory

Your project structure should look something like this:

project-root/reklayer/python/PIL
project-root/reklayer/python/Pillow-8.1.0.dist-info
project-root/reklayer/python/Pillow.libs
  1. Remove the python.zip file to clean up
  2. Terminate the Amazon EC2 Instance that you created to build the archive

Getting Started

  1. npm install

  2. cdk deploy

A 'cdk deploy' will deploy everything that you need into your account

  1. You may now test the backend by uploading an image into your Amazon S3 bucket.

Prerequisites for Episode 7

In episode 7, we are building a deployment pipeline for our application. Before we start working on our pipeline, there are a few things to point out. For this tutorial, you will need:

  1. Github account
  2. Github personal access token. Token should have the scopes repo and admin:repo_hook
  3. Github owner, repository name, branch name set up in AWS Systems Manager - Parameter Store
  4. Github personal access token set up in AWS Secrets Manager

Parameter Store Examples

  • devhour-backend-git-repo Value: aws-dev-hour-backend
  • devhour-backend-git-branch Value: main (Or whichever branch you would like your webhook)
  • devhour-backend-git-owner Value: your-github-username

Episode 7 Notes

In this tutorial, we are using @aws-cdk/pipelines module to build a deployment pipeline. As of 11 March 2021, @aws-cdk/pipelines module is in Developer Preview. So you will need to set a feature flag in cdk.json as below to use new features of the CDK framework.

"@aws-cdk/core:newStyleStackSynthesis": "true"

You will also need to bootstrap the stack again to accommodate the new CDK pipeline experience by running this command:

cdk bootstrap

If you have deployed the stack already into your account using 'cdk deploy', you may need to destroy your stack so that the pipeline can build a fresh one. You can do this using the following command:

cdk destroy AwsdevhourStack

Once you have done this, you can deploy your pipeline stack by running the following command:

cdk deploy AwsdevhourBackendPipelineStack

While deploying, AWS CodePipeline will create a webhook with your Github repo. Subsequent pushes into your repo branch will update your stack automatically, even the pipeline will self-mutate.

To view your available stacks, you can run:

cdk list

** Pillow Library Note **

In awsdevhour-backend-pipeline-stack.ts you will notice the following:

      synthAction: SimpleSynthAction.standardNpmSynth({
        sourceArtifact,
        cloudAssemblyArtifact,
        //This build command is to download pillow library, unzip the downloaded file and tidy up.
        //If you already have pillow library downloaded under reklayer/, please just run 'npm run build'
        buildCommand: 'rm ./reklayer/pillow-goes-here.txt && wget https://awsdevhour.s3-accelerate.amazonaws.com/pillow.zip && unzip pillow.zip && mv ./python ./reklayer && rm pillow.zip && npm run build',
        synthCommand: 'npm run cdk synth'
      })

If you already have the Pillow library under reklayer/python, then you don't need to run this. We added it to facilitate testing during the show. You could simply do the following:

      synthAction: SimpleSynthAction.standardNpmSynth({
        sourceArtifact,
        cloudAssemblyArtifact,
        buildCommand: 'npm run build',
        synthCommand: 'npm run cdk synth'
      })

RekLayer

In this tutorial, we are using Python Imaging Library to add image processing capabilities to our application. As part of the deployment, you can manually download the pillow library and keep under /reklayer folder in this project. So, you can keep the build simple.

However, you can also download the pillow library when you set up the pipeline as shown in awsdevhour-backend-pipeline-stack.ts

Cleanup

To clean up the resources created by the CDK, run the following commands:

aws s3 rm --recursive s3://{imageBucket}
cdk destroy

(Enter “y” in response to: Are you sure you want to delete (y/n)?).

Tweaks

Rekognition confidence is currently set in the rekognition lambda.

minConfidence = 50

Feel free to adjust and experiment. If you change, make sure to perform another 'cdk deploy' to update the lambda function.

Contributions

We would encourage all of our AWS Dev Hour viewers to contribute to this project. For more details, please refer to 'CONTRIBUTING.md'.

License

This software is licensed under the Apache License, Version 2.0.

aws-dev-hour-backend's People

Contributors

amazon-auto avatar maykyaw avatar newtondigital avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

aws-dev-hour-backend's Issues

No module named 'PIL'

Following the lecture, when I uploaded an image to s3 bucket using the CLI, I check the cloudwatch for the logs and noticed that there was a runtime error related to PIL module not being found:

[ERROR] Runtime.ImportModuleError: Unable to import module 'index': No module named 'PIL'

and scanning the dynamodb did not return any item.

cdk destroy error

This is a reopening of #11 where the reporter posted a solution.

However, the solution in #11 was not taught in the twitch stream Episode 1 and did not occur during the destroy on that stream.

Assuming that this error occurs because some addition S3 options have been added into the Episode 1 repo, versus what was streamed, is there a 'fix' that can be added into the code to stop this error?

Unable to import module 'index': No module named 'PIL'

I followed the steps mentioned to create PIL module and setup in my workspace. However, I am getting error in my lambda.

{
"errorMessage": "Unable to import module 'index': No module named 'PIL'",
"errorType": "Runtime.ImportModuleError"
}

After Module 5: `Cannot uploading file: No credentials`

I have deployed to AWS S3 Static, and tried to signup and login all works fine. I am not able to upload file. Also, if I upload it manually using aws cli unable to see in the website.. just assuming it is due to the cognito user that I am logged in.
Screen Shot

cdk destroy error

When I executed the cdk destroy command I encountered this error:

❌ AwsdevhourStack: destroy failed Error: The stack named AwsdevhourStack is in a failed state. You may need to delete it from the AWS console : DELETE_FAILED (The following resource(s) failed to delete: [cdkreknimgagebucketABE44D40]. )
at Object.waitForStackDelete (/Users/FranciscoFranco/.nvm/versions/node/v12.6.0/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:277:11)
at processTicksAndRejections (internal/process/task_queues.js:85:5)
at Object.destroyStack (/Users/FranciscoFranco/.nvm/versions/node/v12.6.0/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:379:28)
at CdkToolkit.destroy (/Users/FranciscoFranco/.nvm/versions/node/v12.6.0/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:252:9)
at initCommandLine (/Users/FranciscoFranco/.nvm/versions/node/v12.6.0/lib/node_modules/aws-cdk/bin/cdk.ts:204:9)
The stack named AwsdevhourStack is in a failed state. You may need to delete it from the AWS console : DELETE_FAILED (The following resource(s) failed to delete: [cdkreknimgagebucketABE44D40]. )

rekognition lambda is not triggered during uploading an image

I've used this repo source code for rekognition lambda and the service lambda and inside the stack I have the same read and write permission to image bucket and dynamoDB as in the source code here. However, when I copy an image to image bucket through CLI, there is neither new entry in the dynamoDB nor in the resized image bucket. Any idea what am I missing? Here is the snippet of my stack:

/**=================================================================
     * Lambda can read from S3 event source when the obj is created in s3
     * =================================================================
     */
    rekFn.addEventSource(new S3EventSource(imageBucket, {
      events: [s3.EventType.OBJECT_CREATED]
    }));
    /**=================================================================
     * Grant s3 read & put permission to lambda
     * =================================================================
     */
    imageBucket.grantRead(rekFn)
    imageBucket.grantPut(rekFn)
    /**=================================================================
     * Grant dynamodb write permission to lambda
     * =================================================================
     */
    table.grantWriteData(rekFn);
    rekFn.addToRolePolicy( new iam.PolicyStatement({
      effect: iam.Effect.ALLOW,
      actions: ['rekognition:DetectLabels'],
      resources: ['*']
    }))    

    /**=================================================================
     * Lambda for Synchronous Front End
     * =================================================================
     */
    const serviceFn = new lambda.Function(this, 'serviceFunction', {
      code: lambda.Code.fromAsset('servicelambda'),
      runtime: lambda.Runtime.PYTHON_3_7,
      handler: 'index.handler',
      environment: {
        "TABLE": table.tableName,
        "BUCKET": imageBucket.bucketName,
        "RESIZEDBUCKET": resizedBucket.bucketName
      }
    })

ENOENT: no such file or directory

Hi, everytime I run any command it gives me this error:
cdk synth -v

ENOENT: no such file or directory, stat '/Users/jedrek/workspace/devhr-project/rekognitionlambda'
Subprocess exited with error 1
Error: Subprocess exited with error 1
    at ChildProcess.<anonymous> (/usr/local/lib/node_modules/aws-cdk/lib/api/cxapp/exec.ts:122:23)
    at ChildProcess.emit (events.js:314:20)
    at ChildProcess.EventEmitter.emit (domain.js:486:12)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:276:12)

I have copy-pasted all the code from your repo to my local app.
cdk bootstrap aws://.../eu-central-1 -v

ENOENT: no such file or directory, stat '/Users/jedrek/workspace/devhr-project/rekognitionlambda'
Subprocess exited with error 1
Error: Subprocess exited with error 1
    at ChildProcess.<anonymous> (/usr/local/lib/node_modules/aws-cdk/lib/api/cxapp/exec.ts:122:23)
    at ChildProcess.emit (events.js:314:20)
    at ChildProcess.EventEmitter.emit (domain.js:486:12)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:276:12)

Why am I getting this error? Any idea?

[ERROR] Runtime.ImportModuleError: Unable to import module 'index': cannot import name '_imaging' from 'PIL' (/opt/python/PIL/__init__.py)

I am trying to deploy the changes for Episode 7 but I am struggling to get it working.
I kept the PIL library in the reklayer/python directory and I updated the pipeline stack with:

      synthAction: SimpleSynthAction.standardNpmSynth({
        sourceArtifact,
        cloudAssemblyArtifact,
        buildCommand: 'npm run build',
        synthCommand: 'npm run cdk synth'
      })

But I get this error in cloudwatch:
[ERROR] Runtime.ImportModuleError: Unable to import module 'index': cannot import name '_imaging' from 'PIL' (/opt/python/PIL/__init__.py)

Here is my main repository: https://github.com/piedpiper-ff4084/aws-dev-hour-backend

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.