Extract specific parts of the UK Met Office's UKV and MOGREPS-UK numerical weather predictions from AWS, compress, and save to S3 as Zarr. Intended to run on AWS EC2.
conda env create -f environment.yml
conda activate metoffice_ec2
pip install -e .
py.test -s
If boto3 is setup to access AWS, then you can run scripts/ec2.py from your local machine to test (although it'll try to pull large amounts of data into & out of S3, so this will get expensive quickly!)
docker build .
docker run -e AWS_ACCESS_KEY_ID=<ID> -e AWS_SECRET_ACCESS_KEY=<KEY> -e DEST_BUCKET=<BUCKET> -e SQS_URL=<SQS_URL> <DockerImage>
A Terraform template is provided in /infrastructure.
A Docker Image is also provided publicly.
> More Information about Infrastructure
The following environment variables must be provided:
| Name | Type | Description |
|---|---|---|
SQS_URL |
String |
The URL of the SQS that the Messages are consumed from |
DEST_BUCKET |
String |
S3 bucket that the finished output should be stored in |
Manual Setup
Go to the AWS Identity and Access Management (IAM) console... TODO
Create a bucket for storing subsetted NWPs. Set the DEST_BUCKET constant in scripts/ec2.py
When the Met Office uploads new NWPs to S3, they also send a message to an AWS Simple Notification Service topic. These notifications must be received as soon as they're produced. But our EC2 job isn't kept running 24/7. Our EC2 job is triggered once an hour. So we need a way to capture the SNS notifications when our EC2 job is offline.
A solution is to set up an AWS Simple Queue Service. Set up SQS as per the Met Office's instructions.
Then set the SQS_URL in scripts/ec2.py
This code follows the Google Python Style Guide.
The included environment.yml file includes just the packages required to run the script
and unit tests.
To do development, please run this command within the metoffice_ec2 conda environment:
conda install black flake8 isort jedi mypy
Format the code with black metoffice_ec2 scripts.
Check coding style with flake8.
Fix import order with isort ..
Run static type checking with mypy metoffice_ec2 scripts.
jedi is for auto-completion (and other things) in IDEs.
Thanks goes to these wonderful people (emoji key):
 Jack Kelly 💻 |
 Flo 💻 |
 Tom White 💻 |
This project follows the all-contributors specification. Contributions of any kind welcome!
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent