Code Monkey home page Code Monkey logo

Comments (7)

sachalau avatar sachalau commented on September 23, 2024 1

Not sure I understood your question, but it was a really lame path for demonstration :

{"params":{"memory":{"bwa":{"at":{"runtime":10000}}}},"vcpus":{"bwa":{"at":{"runtime":8}}}}

Are you sure you want to define these requirements at run time though ? is it usual that you need different resources for these jobs ?

from genomics-secondary-analysis-using-aws-step-functions-and-aws-batch.

sachalau avatar sachalau commented on September 23, 2024 1

Almost :

BatchJobDefinitionFastp:
  Type: "AWS::Batch::JobDefinition"
  Properties:
    JobDefinitionName: !Sub ${ProjectLowerCase}-fastp
    Type: container #required
    ContainerProperties:
      Image: !Sub ${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${ProjectLowerCase}-fastp
      ResourceRequirements:
         - Type: VCPU
           Value: 8
         - Type: MEMORY
           Value: 16000
      Volumes:
        - Host:
            SourcePath: /opt/miniconda
          Name: awscli
      MountPoints:
        - ContainerPath: /opt/miniconda
          SourceVolume: awscli

https://docs.aws.amazon.com/fr_fr/AWSCloudFormation/latest/UserGuide/aws-properties-batch-jobdefinition-resourcerequirement.html

Edited because I answered too rapidly.

from genomics-secondary-analysis-using-aws-step-functions-and-aws-batch.

sachalau avatar sachalau commented on September 23, 2024

That's something you have to define when you submit a batch job.

https://docs.aws.amazon.com/step-functions/latest/dg/connect-batch.html

You must define the values in the StepFunction JSON/YAML, under containersOverrides (https://docs.aws.amazon.com/batch/latest/APIReference/API_SubmitJob.html#Batch-SubmitJob-request-containerOverrides) then resourcesRequirements (https://docs.aws.amazon.com/batch/latest/APIReference/API_ResourceRequirement.html) for each batch job definition.

Do not define them under containerOverrides, vcpus and memory directly because those have been deprecated.

Actually you must edit all the batch job definitions in main.yaml because these have been defined with vcpus and memory directly, which are now deprecated. I think if you do not do that, the values you try to pass will be overriden.

from genomics-secondary-analysis-using-aws-step-functions-and-aws-batch.

spitfiredd avatar spitfiredd commented on September 23, 2024

So is there any way to override the default from your workflow JSON/YAML at runtime?

from genomics-secondary-analysis-using-aws-step-functions-and-aws-batch.

sachalau avatar sachalau commented on September 23, 2024

Sure :

 "BwaMem": {
          "Type": "Task",
          "ResultPath": null,
          "Resource": "arn:aws:states:::batch:submitJob.sync",
          "Parameters": {
              "JobName": "bwa-mem",
              "JobDefinition.$": "$.jobdefs.bwa",
              "JobQueue.$": "$.params.queue",
              "ContainerOverrides": {
                    "ResourceRequirements": [
                        {
                            "Type": "VCPU",
                            "Value.$": "$.params.vcpus.bwa.at.runtime"
                        },
                        {
                            "Type": "MEMORY",
                            "Value.$": "$.params.memory.bwa.at.runtime"
                        }
                    ],
                  "Command": [
                      "bwa",
                      "mem",
                      "-R",
                      "Ref::HEADER",
                      "-t",
                      "4",
                      "-o",
                      "Ref::SAM_FILE",
                      "Ref::REFERENCE",
                      "Ref::READ_R1",
                      "Ref::READ_R2"
                  ]
              },
              "Parameters": {
                  "HEADER.$": "States.Format('@RG\\\\tID:{}\\\\tSM:{}\\\\tPL:ILLUMINA', $.FastqId, $.SampleId)",
                  "SAM_FILE.$": "States.Format('{}/{}.sam', $.SampleId, $.FastqId)",
                  "REFERENCE.$": "States.Format('references/{}.fna', $.REFERENCE_NAME)",
                  "READ_R1.$": "States.Format('{}/{}_{}', $.SampleId, $.FastqId, $.reads.suffix_R1)",
                  "READ_R2.$": "States.Format('{}/{}_{}', $.SampleId, $.FastqId, $.reads.suffix_R2)"
              }
          },
          "Next": "SamtoolsSort"
      }

from genomics-secondary-analysis-using-aws-step-functions-and-aws-batch.

spitfiredd avatar spitfiredd commented on September 23, 2024

@sachalau - with this configuration; how do you start the state machine?

from genomics-secondary-analysis-using-aws-step-functions-and-aws-batch.

spitfiredd avatar spitfiredd commented on September 23, 2024

@sachalau - one more question about this. I updated the ResourceRequirements as you suggested and when I run the state machine with the memory/vcpu params, I get still get a deprecation warning This value was submitted using containerOverrides.vcpus which has been deprecated and was not used as an override. Instead, the VCPU value found in the job definition’s resourceRequirements key was used instead. More information about the deprecated key can be found in the AWS Batch API documentation. Learn more

I am thinking I also need to update the batch job definition?

Currently it's,

BatchJobDefinitionFastp:
  Type: "AWS::Batch::JobDefinition"
  Properties:
    JobDefinitionName: !Sub ${ProjectLowerCase}-fastp
    Type: container #required
    ContainerProperties:
      Image: !Sub ${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${ProjectLowerCase}-fastp
      Vcpus: 8
      Memory: 16000
      Volumes:
        - Host:
            SourcePath: /opt/miniconda
          Name: awscli
      MountPoints:
        - ContainerPath: /opt/miniconda
          SourceVolume: awscli

Would I need to update this as well? I was thinking something like,

BatchJobDefinitionFastp:
  Type: "AWS::Batch::JobDefinition"
  Properties:
    JobDefinitionName: !Sub ${ProjectLowerCase}-fastp
    Type: container #required
    ContainerProperties:
      Image: !Sub ${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${ProjectLowerCase}-fastp
      ResourceRequirements:
          Vcpus: 8
          Memory: 16000
      Volumes:
        - Host:
            SourcePath: /opt/miniconda
          Name: awscli
      MountPoints:
        - ContainerPath: /opt/miniconda
          SourceVolume: awscli

from genomics-secondary-analysis-using-aws-step-functions-and-aws-batch.

Related Issues (12)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.