Comments (7)
Not sure I understood your question, but it was a really lame path for demonstration :
{"params":{"memory":{"bwa":{"at":{"runtime":10000}}}},"vcpus":{"bwa":{"at":{"runtime":8}}}}
Are you sure you want to define these requirements at run time though ? is it usual that you need different resources for these jobs ?
from genomics-secondary-analysis-using-aws-step-functions-and-aws-batch.
Almost :
BatchJobDefinitionFastp:
Type: "AWS::Batch::JobDefinition"
Properties:
JobDefinitionName: !Sub ${ProjectLowerCase}-fastp
Type: container #required
ContainerProperties:
Image: !Sub ${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${ProjectLowerCase}-fastp
ResourceRequirements:
- Type: VCPU
Value: 8
- Type: MEMORY
Value: 16000
Volumes:
- Host:
SourcePath: /opt/miniconda
Name: awscli
MountPoints:
- ContainerPath: /opt/miniconda
SourceVolume: awscli
Edited because I answered too rapidly.
from genomics-secondary-analysis-using-aws-step-functions-and-aws-batch.
That's something you have to define when you submit a batch job.
https://docs.aws.amazon.com/step-functions/latest/dg/connect-batch.html
You must define the values in the StepFunction JSON/YAML, under containersOverrides (https://docs.aws.amazon.com/batch/latest/APIReference/API_SubmitJob.html#Batch-SubmitJob-request-containerOverrides) then resourcesRequirements (https://docs.aws.amazon.com/batch/latest/APIReference/API_ResourceRequirement.html) for each batch job definition.
Do not define them under containerOverrides, vcpus and memory directly because those have been deprecated.
Actually you must edit all the batch job definitions in main.yaml because these have been defined with vcpus and memory directly, which are now deprecated. I think if you do not do that, the values you try to pass will be overriden.
from genomics-secondary-analysis-using-aws-step-functions-and-aws-batch.
So is there any way to override the default from your workflow JSON/YAML at runtime?
from genomics-secondary-analysis-using-aws-step-functions-and-aws-batch.
Sure :
"BwaMem": {
"Type": "Task",
"ResultPath": null,
"Resource": "arn:aws:states:::batch:submitJob.sync",
"Parameters": {
"JobName": "bwa-mem",
"JobDefinition.$": "$.jobdefs.bwa",
"JobQueue.$": "$.params.queue",
"ContainerOverrides": {
"ResourceRequirements": [
{
"Type": "VCPU",
"Value.$": "$.params.vcpus.bwa.at.runtime"
},
{
"Type": "MEMORY",
"Value.$": "$.params.memory.bwa.at.runtime"
}
],
"Command": [
"bwa",
"mem",
"-R",
"Ref::HEADER",
"-t",
"4",
"-o",
"Ref::SAM_FILE",
"Ref::REFERENCE",
"Ref::READ_R1",
"Ref::READ_R2"
]
},
"Parameters": {
"HEADER.$": "States.Format('@RG\\\\tID:{}\\\\tSM:{}\\\\tPL:ILLUMINA', $.FastqId, $.SampleId)",
"SAM_FILE.$": "States.Format('{}/{}.sam', $.SampleId, $.FastqId)",
"REFERENCE.$": "States.Format('references/{}.fna', $.REFERENCE_NAME)",
"READ_R1.$": "States.Format('{}/{}_{}', $.SampleId, $.FastqId, $.reads.suffix_R1)",
"READ_R2.$": "States.Format('{}/{}_{}', $.SampleId, $.FastqId, $.reads.suffix_R2)"
}
},
"Next": "SamtoolsSort"
}
from genomics-secondary-analysis-using-aws-step-functions-and-aws-batch.
@sachalau - with this configuration; how do you start the state machine?
from genomics-secondary-analysis-using-aws-step-functions-and-aws-batch.
@sachalau - one more question about this. I updated the ResourceRequirements as you suggested and when I run the state machine with the memory/vcpu params, I get still get a deprecation warning This value was submitted using containerOverrides.vcpus which has been deprecated and was not used as an override. Instead, the VCPU value found in the job definition’s resourceRequirements key was used instead. More information about the deprecated key can be found in the AWS Batch API documentation. Learn more
I am thinking I also need to update the batch job definition?
Currently it's,
BatchJobDefinitionFastp:
Type: "AWS::Batch::JobDefinition"
Properties:
JobDefinitionName: !Sub ${ProjectLowerCase}-fastp
Type: container #required
ContainerProperties:
Image: !Sub ${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${ProjectLowerCase}-fastp
Vcpus: 8
Memory: 16000
Volumes:
- Host:
SourcePath: /opt/miniconda
Name: awscli
MountPoints:
- ContainerPath: /opt/miniconda
SourceVolume: awscli
Would I need to update this as well? I was thinking something like,
BatchJobDefinitionFastp:
Type: "AWS::Batch::JobDefinition"
Properties:
JobDefinitionName: !Sub ${ProjectLowerCase}-fastp
Type: container #required
ContainerProperties:
Image: !Sub ${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${ProjectLowerCase}-fastp
ResourceRequirements:
Vcpus: 8
Memory: 16000
Volumes:
- Host:
SourcePath: /opt/miniconda
Name: awscli
MountPoints:
- ContainerPath: /opt/miniconda
SourceVolume: awscli
from genomics-secondary-analysis-using-aws-step-functions-and-aws-batch.
Related Issues (12)
- Issues when executing build-s3-dist.sh HOT 6
- Multiple Input files have the same name.
- 403 downloading s3 assets HOT 1
- lacking lambda permissions for Cloudformation Zone role?
- Error while downloading sample files HOT 2
- Deploy customized solution HOT 10
- [Errno 28] No Space left on device HOT 2
- Deployment fails at setup in the stack formation HOT 2
- Error building when modifying the solution HOT 6
- Where in the code can I increase the size of the persistant EC2 volume? HOT 2
- No Space left on device HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from genomics-secondary-analysis-using-aws-step-functions-and-aws-batch.