aws-samples / annotate-medical-images-in-dicom-server-and-build-ml-models-on-amazon-sagemaker Goto Github PK

View Code? Open in Web Editor NEW

19.0 10.0 18.0 40.42 MB

License: MIT No Attribution

Dockerfile 0.99% Shell 9.94% HTML 19.51% Jupyter Notebook 23.40% Python 8.86% CSS 21.54% JavaScript 15.77%

annotate-medical-images-in-dicom-server-and-build-ml-models-on-amazon-sagemaker's Introduction

Annotate Medical Images in DICOM Server and Build ML models using Amazon SageMaker

Overview Architecture

Deployment Steps

Pre-requisite

You will need two public subnets and two private subnets in a given Virtual Private Cloud (VPC). You can either follow this blog post to deploy the network infrastructure first, or deploy the VPC network infrastructure using this CloudFormation template (Optional)

Deploy Orthanc DICOM Server on AWS ECS

To deploy the CloudFormation (CFN) on AWS console, using 1-click deployment button:

The required information will be EC2 Key pair and network infrastructure.

After the CFN stack is successfully created, the Orthanc endpoint URL will be available in the Outputs tab like:

Upload DICOM images to Orthanc

Orthanc supports WADO-RS with its RESTful API. You can upload a DICOM image from local folder:

curl -u orthanc:orthanc -X POST https://<Orthanc URL>/instances --data-binary @<DICOM Image>.dcm -k

After uploaded, check the instance id assigned: curl -u orthanc:orthanc https://<Orthanc URL>/instances -k

Then put instance IDs in source field of manifest.json. Assuming here is the output of the previous command (list of DICOM instances): [ "502b0a4b-5cb43965-7f092716-bd6fe6d6-4f7fc3ce", "af163634-5f8009c8-a941f2a2-4d108bea-119181b5", "3e9562ab-3bd7b304-e003d1f8-0fe2e8f5-e6af3f36" ]
The manifest.json file will look at this.

Deploy the SageMaker Labeling and Notebook Resources

click the button below to deploy lambda functions, S3 bucket, and Notebook instance with appropriate IAM roles:

Fill the parameter PreLabelLambdaSourceEndpointURL with the Orthanc endpoint URL from previous step.

Take a note of the outputs of CFN deployment, including SMGTLabelingExecutionRole and SageMakerAnnotationS3Bucket names:

The manifest.json file generated earlier will be uploaded the the SageMakerAnnotationS3Bucket.

Create Custom Label Job in SageMaker GroundTruth

Upload the input manifest.json file generated earlier to the SageMakerAnnotationS3Bucket
Create workforce and add team member
Create a SageMaker GroundTruth Labeling Job: at step 1 specify the input and output data locations in SageMakerAnnotationS3Bucket plus the SMGTLabelingExecutionRole as IAM role for labeling job
Copy the content in template.liquid.html to the Custom Template text field as well as configure the gt-prelabel-task-lambda and gt-postlabel-task-lambda functions deployed earlier.

After configuring the custom labeling task, click on the Preview button, you will see the following preview

If you created a private workforce, you can go to the Labeling Workforces tab and find the annotation console link there as shown earlier. The annotator will be able to see the task created like:

Train and deploy model using SageMaker Notebook Instance

The CFN deployment earlier also create a SageMaker notebook instance, open the Jupyter notebook to train and deploy machine learning models using the annotated datasets from SageMaker GroundTruth labeling job:

Security

See CONTRIBUTING for more information.

License

This library is licensed under the MIT-0 License. See the LICENSE file.

annotate-medical-images-in-dicom-server-and-build-ml-models-on-amazon-sagemaker's People

Contributors

Stargazers

Watchers

Forkers

chafey gabeta sheriefemam georgepapaioannou miguelsotoc masazi hmesha athewsey edsun3941 devhliu danullom stoviamed craigspaz fjkiani

annotate-medical-images-in-dicom-server-and-build-ml-models-on-amazon-sagemaker's Issues

CloudFormation stack to deploy lambda functions, S3 bucket, and Notebook instance with appropriate IAM roles is not accessible.

link: S3 URL: https://orthanc-on-aws.s3.amazonaws.com/sagemaker_resources.yaml

error: S3 error: Access Denied For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html

[Feature] Support label adjustment / review workflow

For a project I've worked on recently, we'd like to implement second-line review of annotations using a DICOM template like this.

It'd be super helpful if the sample template provided here could also be used for adjustment of pre-existing labels, as supported by some of the built-in task UIs. This way, a second SMGT job could review and amend the annotations from the initial one.

I've already taken an initial look at how this could be done & will raise a draft PR - seems like it shouldn't require too much change, but there are a couple of points where there might be broader impact to the sample thought it'd be best to check in before trying too hard!

Cloud formation stack creation error

Resource template validation failed for resource OrthancTaskDefinition as the template has invalid properties. Please refer to the resource documentation to fix the template. Properties validation failed for resource OrthancTaskDefinition with message: #/ContainerDefinitions/0/LinuxParameters: extraneous key [initProcessEnabled] is not permitted

ORTHANC server setup cloud formation stack is not assessable

link https://orthanc-on-aws.s3.amazonaws.com/orthanc-ec2-rds-cfn-tempalte.yaml is not assessable, access denied .

Access denied to some of the Sagemaker Resource files when accessed via our IAM roles

I am trying to reuse this AWS-SAMPLES and this is blocking my work from past few weeks:
https://github.com/aws-samples/annotate-medical-images-in-dicom-server-and-build-ml-models-on-amazon-sagemaker

This XML file does not appear to have any style information associated with it. The document tree is shown below.

AccessDenied
Access Denied
XKAXHCS704WHF1WC
vV5RoB95rqj6qN4pYItCCwItOkUWMwLIOwYUMgTaW37UQX1a6ZUpk5dwgQzjO21OpPDGNVsC+Qg=

CORS issue when launching the labelling taks

Hi,
Not sure I can post this as an issue, pretty new to GitHub... sorry if I should not have.

I have an up and running Orthanc on AWS so I do not use the stack to create it.
Mine runs on an EC2 instance, ubuntu, behing an elastic load balancer, using S3 and Aurora PostgreSQL serverless. It works.
I use the CF template here to deploy the lambda function, roles, bucket...

When I start the labelling tool, I have this error:
Access to XMLHttpRequest at 'http://my-elb-dns-name-goes-here/instances/32de62c1-381ac837-e843f0ae-770d263f-b925bb4c/file' from origin 'https://s3.amazonaws.com' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.

I have tried many things, like CORS policy on the Orthanc used S3 bucket, the labelling S3 bucket. I have installed a NGNIX reverse proxy between the ELB and Orthanc to configure this: https://book.orthanc-server.com/faq/nginx.html#nginx

So I am not sure why I get this error. Is there something specific that I might have missed, that is configured in the template provided here for Orthanc.

Thanks
Julien