Run a Python script that processes and exports data from external sources to a Google Sheet on AWS Lambda, deployed via AWS ECR.
- Python installed on local machine
- Poetry installed on local machine (e.g. via pip)
- docker desktop installed on local machine
- Google Cloud access to create service account
- AWS access to push to ECR and create Lambda functions
- Initialize new package
mkdir lambda-workshop
poetry init
git init
-
Create/copy lambda_function.py, modules/sheet.py, modules/finance.py and .gitginore
-
Create Google Cloud project and service account, generate and download key as JSON file, add file to root folder as google_account.json
-
Create Google Sheet, add edit permission for service account and update sheet ID in sheet.py
-
Add dependencies via Poetry
poetry add yahooquery pandas google-api-python-client
- Test script locally
poetry run python -c "import lambda_function as lam; lam.lambda_handler()"
-
Create/copy Dockerfile
-
Create AWS ECR repository
-
Create AWS Lambda function with "lambda-test" execution role and 5 minutes timeout
-
Follow deployment steps below
Create virtual environment and install dependencies with Poetry
poetry install
poetry run python -c "import lambda_function as lam; lam.lambda_handler()"
- Authenticate with ECR (expects configured AWS CLI)
aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin 880172922216.dkr.ecr.eu-west-1.amazonaws.com
- Build docker image
docker build -t lambda-workshop . --platform linux/amd64
- Tag image
docker tag lambda-workshop:latest 880172922216.dkr.ecr.eu-west-1.amazonaws.com/lambda-workshop:latest
- Push image
docker push 880172922216.dkr.ecr.eu-west-1.amazonaws.com/lambda-workshop:latest
- Update lambda function
aws lambda update-function-code --region eu-west-1 --function-name lambda-workshop --image-uri 880172922216.dkr.ecr.eu-west-1.amazonaws.com/lambda-workshop:latest
- Set lambda environment variables (from locally sourced environment)
aws lambda update-function-configuration --region eu-west-1 --function-name lambda-workshop --environment "Variables={SYMBOL=$SYMBOL}"
- (Optional) Schedule cron job to trigger function execution