Comments (2)
drdivine
commented on September 23, 2024
1
Just a follow up. It seems that there are some errors in the userdata script in the launch template. could someone verify it does the right thing?
UserData:
Fn::Base64: |
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==BOUNDARY=="
--==BOUNDARY==
Content-Type: text/cloud-config; charset="us-ascii"
packages:
- jq
- btrfs-progs
- python27-pip
- sed
- wget
- git
- bzip2
- amazon-ssm-agent
runcmd:
- pip install -U awscli boto3
- start amazon-ssm-agent
- stop ecs
- service docker stop
# install amazon-ebs-autoscale
- cp -au /var/lib/docker /var/lib/docker.bk
- rm -rf /var/lib/docker/*
- EBS_AUTOSCALE_VERSION=$(curl --silent "https://api.github.com/repos/awslabs/amazon-ebs-autoscale/releases/latest" | jq -r .tag_name)
- cd /opt && git clone https://github.com/awslabs/amazon-ebs-autoscale.git
- cd /opt/amazon-ebs-autoscale && git checkout $EBS_AUTOSCALE_VERSION
- sh /opt/amazon-ebs-autoscale/install.sh /var/lib/docker /dev/sdc 2>&1 > /var/log/ebs-autoscale-install.log
- sed -i 's+OPTIONS=.*+OPTIONS="--storage-driver btrfs"+g' /etc/sysconfig/docker-storage
- cp -au /var/lib/docker.bk/* /var/lib/docker
# install miniconda/awscli
- wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
- bash Miniconda3-latest-Linux-x86_64.sh -b -f -p /opt/miniconda
- /opt/miniconda/bin/conda install -c conda-forge -y awscli
- chown -R ec2-user:ec2-user /opt/miniconda
- rm Miniconda3-latest-Linux-x86_64.sh
- service docker start
- start ecs
--==BOUNDARY==--This is also different then what is used in the genomics-workflow repository:
additions: |-
- stop ecs
- service docker stop
- cp -au /var/lib/docker /var/lib/docker.bk
- rm -rf /var/lib/docker/*
- cd /opt && wget $artifactRootUrl/get-amazon-ebs-autoscale.sh
- sh /opt/get-amazon-ebs-autoscale.sh
- sh /opt/amazon-ebs-autoscale/install.sh $scratchPath /dev/sdc > /var/log/ebs-autoscale-install.log 2>&1
- sed -i 's+OPTIONS=.*+OPTIONS="--storage-driver btrfs"+g' /etc/sysconfig/docker-storage
- cp -au /var/lib/docker.bk/* /var/lib/docker
- cd /opt && wget $artifactRootUrl/aws-ecs-additions.tgz && tar -xzf aws-ecs-additions.tgz
- sh /opt/ecs-additions/ecs-additions-step-functions.sh
- service docker start
- start ecs
and
UserData:
Fn::Base64: !Sub
- |
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==BOUNDARY=="
--==BOUNDARY==
Content-Type: text/cloud-config; charset="us-ascii"
packages:
- jq
- btrfs-progs
- sed
- wget
- git
- amazon-ssm-agent
- unzip
runcmd:
- curl -s "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "/tmp/awscliv2.zip"
- unzip -q /tmp/awscliv2.zip -d /tmp
- /tmp/aws/install
- export scratchPath="${ScratchMountPoint}"
- export artifactRootUrl="${ArtifactRootUrl}"
- start amazon-ssm-agentI'm getting the following errors when I install by hand:
+ MOUNTPOINT=/scratch
+ SIZE=100
+ DEVICE=
+ FILE_SYSTEM=btrfs
++ dirname /opt/amazon-ebs-autoscale/install.sh
+ BASEDIR=/opt/amazon-ebs-autoscale
+ . /opt/amazon-ebs-autoscale/shared/utils.sh
+ initialize
++ curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone/
+ export AWS_AZ=eu-west-1a
+ AWS_AZ=eu-west-1a
++ echo eu-west-1a
++ sed -e 's/[a-z]$//'
+ export AWS_REGION=eu-west-1
+ AWS_REGION=eu-west-1
++ curl -s http://169.254.169.254/latest/meta-data/instance-id
+ export INSTANCE_ID=i-029e03fc7d0eb2f14
+ INSTANCE_ID=i-029e03fc7d0eb2f14
+ export EBS_AUTOSCALE_CONFIG_FILE=/etc/ebs-autoscale.json
+ EBS_AUTOSCALE_CONFIG_FILE=/etc/ebs-autoscale.json
+ PARAMS=
+ (( 2 ))
+ case "$1" in
+ PARAMS=' /var/lib/docker'
+ shift
+ (( 1 ))
+ case "$1" in
+ PARAMS=' /var/lib/docker /dev/sdc'
+ shift
+ (( 0 ))
+ eval set -- ' /var/lib/docker /dev/sdc'
++ set -- /var/lib/docker /dev/sdc
+ '[' '!' -z ' /var/lib/docker /dev/sdc' ']'
+ MOUNTPOINT=/var/lib/docker
+ '[' '!' -z /dev/sdc ']'
+ DEVICE=/dev/sdc
+ mkdir -p /usr/local/amazon-ebs-autoscale/bin /usr/local/amazon-ebs-autoscale/shared
+ cp /opt/amazon-ebs-autoscale/bin/create-ebs-volume /opt/amazon-ebs-autoscale/bin/ebs-autoscale /usr/local/amazon-ebs-autoscale/bin
+ chmod +x /usr/local/amazon-ebs-autoscale/bin/create-ebs-volume /usr/local/amazon-ebs-autoscale/bin/ebs-autoscale
+ ln -sf /usr/local/amazon-ebs-autoscale/bin/create-ebs-volume /usr/local/amazon-ebs-autoscale/bin/ebs-autoscale /usr/local/bin/
+ ln -sf /usr/local/amazon-ebs-autoscale/bin/create-ebs-volume /usr/local/amazon-ebs-autoscale/bin/ebs-autoscale /usr/bin/
+ cp /opt/amazon-ebs-autoscale/shared/utils.sh /usr/local/amazon-ebs-autoscale/shared
+ cp /opt/amazon-ebs-autoscale/config/ebs-autoscale.logrotate /etc/logrotate.d/ebs-autoscale
+ cat /opt/amazon-ebs-autoscale/config/ebs-autoscale.json
+ sed -e s#%%MOUNTPOINT%%#/var/lib/docker#
+ sed -e s#%%FILESYSTEM%%#btrfs#
+ '[' -e /var/lib/docker ']'
+ '[' -d /var/lib/docker ']'
+ '[' -e /var/lib/docker ']'
+ '[' -z /dev/sdc ']'
+ '[' '!' -b /dev/sdc ']'
++ create-ebs-volume --size 100
/usr/bin/create-ebs-volume: line 169: aws: command not found
/usr/bin/create-ebs-volume: line 175: aws: command not found
/usr/bin/create-ebs-volume: line 183: aws: command not found
/usr/bin/create-ebs-volume: line 185: [: : integer expression expected
/usr/bin/create-ebs-volume: line 190: [: : integer expression expected
/usr/bin/create-ebs-volume: line 195: [: : integer expression expected
Error: could not create volume
+ DEVICE='/usr/bin/create-ebs-volume: line 218: aws: command not found'
Any help would be greatly appreciated.
from genomics-secondary-analysis-using-aws-step-functions-and-aws-batch.
wleepang
commented on September 23, 2024
1
Hi @drdivine ,
If you still have the instance available, can you check /var/log/cloud-init-output.log for any errors that may have occurred while running the LaunchTemplate steps?
This solution and genomics-workfows are related projects and will eventually have aligned codebases. The core steps of the UserData sections are roughly the same:
- Install requisite system packages (i.e. jq, btrfs-progs)
- Install the AWS CLI
- Provision the instance - e.g. get / install amazon-ebs-autoscale, run any orchestrator specific provisioning steps.
Can you provide some additional context for how you ran the manual install of amazon-ebs-autoscale? What AMI did you use? The errors at the bottom indicate that the AWS CLI was not properly installed. Amazon EBS Autoscale uses the AWS CLI to create and attach volumes.
from genomics-secondary-analysis-using-aws-step-functions-and-aws-batch.
Related Issues (12)
- Issues when executing build-s3-dist.sh HOT 6
- Multiple Input files have the same name.
- 403 downloading s3 assets HOT 1
- lacking lambda permissions for Cloudformation Zone role?
- Error while downloading sample files HOT 2
- Deploy customized solution HOT 10
- Deployment fails at setup in the stack formation HOT 2
- Error building when modifying the solution HOT 6
- Is there any way to increase the VCPU and Memory when starting a state machine? HOT 7
- Where in the code can I increase the size of the persistant EC2 volume? HOT 2
- No Space left on device HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from genomics-secondary-analysis-using-aws-step-functions-and-aws-batch.