data2health / covid19-challenge Goto Github PK

  return {site: "UW",
          dataset_name: "uw_omop_covid_q1",
          train_dataset_version: "05-06-2020",
          infer_dataset_version: "05-06-2020"};

https://www.synapse.org/#!Synapse:syn22093564/tables/

• Pull new sagebionetworks/synapse-workflow-orchestrator-toil:latest
• Use CWL spec v1.1 to set a time limit on docker run times

Pages should be created under

• Results
• Dashboard
• Internal

• Preparation (Thomas)
  • Update example to make sure that the tutorial is easy to follow
  • Create thumbnail for the video (to be used as the video preview)
  • Create spec for the video (max duration, fps, resolution, etc.)
• Make video tutorial (Verena)

• We could highlight this paragraph by showing it in a yellow box.
• Link to challenge rules may need to be remove or use custom version of the rules

• Add ITHS logo

Places that need to be editted for dataset name + dataset version

• create_internal_dataset_version
• create_dataset_version
• get_site.cwl

Sage-Bionetworks/SynapseWorkflowOrchestrator#22

Currently using a unmerged branch in challengeworkflowtemplate. Use new tagged version once merged.

Catch the prediction file empty exception instead of throwing an error and automatically edit the UW submission status column

https://www.synapse.org/#!Synapse:syn21849256/wiki/601866

Important: Do not post sensitive information about the datasets here (public GitHub repo)

Current if a submission fails internally, the error doesn't always get propagated back.

https://www.synapse.org/#!Synapse:syn21849256/wiki/601866

The datasets that are use in the run_train and run_docker steps must be in docker volumes

• when making the registration page available
• when starting accepting submission
• etc.

• Creates Queues
  (Main, Main TEST, Site internal, Site internal TEST)
• Sets correct permissions
• Creates entity with linking workflows
• Makes a copy of the relevant workflows

• Update the FAQ section a last time before the launch of the challenge

FAQ is draft in this Google doc

Add a section on the top of the home page to advertise the latest news.

Remove content and add pointer to useful resources (Justing mentioned a bioRxiv page)

• need to map out goldstandard files as well.

• Identify date of the webinar
• Prepare the webinar (slides, demo, etc.)

See Challenge Questions.

Tasks

• List the requirements that must be met to open Question 2-5 (Yao)
• Status update for Question 2-5

We will reuse the same server as for the Patient Mortality challenge. Will we get additional servers funded by CD2H or other sources of funding?

Justin mentioned that IBM and X may be interested in helping, but if it's cloud servers UW may not be willing to place EHR data there.

synthetic_omop_covid_q1
05-15-2020

Loop in Hsiao-Ching and Julie (already done) to prepare the press release. Share available material with them.

• Create registration page: https://www.synapse.org/#!Synapse:syn21849255
• Add content to the page
• Make the page publicly accessible

eval_dataset_name = "uw_omop_covid_eval"
eval_dataset_version = "05-06-2020"

train_dataset_name = "uw_omop_covid_train"
train_dataset_version = "05-06-2020"

Use these values to create the input_dir in workflow.

• Tom to create and configure submission queue "COVID-19 DREAM Challenge: Question 1"
• Thomas to reorg baseline method GH repo to store examples for different questions

• Identify and document datasets #4
• Identify and document the scoring metrics #5
• Create a page that aggregates COVID-19 related resources #6
• Create challenge Home page #10
• Create How to participate page #11
• Create submissions queues
• Create submissions dashboard
• Create leaderboard tables

Final steps

• Mirror wiki to production site
• Make production site publicly accessible

During our weekly call on Monday, we will test

Initial test

• Test submission to Challenge Question 1 - UW Dataset - version 1
  • Submission successfully evaluated
  • Sub shows up in Dashboard
  • Sub shows up in Leaderboard for Challenge Question 1 - UW Dataset - version 1

Create new Challenge Question 2 - UW Dataset Y* - version 1

• Test submission to Challenge Question 2 - UW Dataset - version 1
  • Submission successfully evaluated
  • Sub shows up in Dashboard
  • Sub shows up in Leaderboard for Challenge Question 2 - UW Dataset - version 1

"Simulate" new UW Dataset version

• Test submission to Challenge Question 1 - UW Dataset - version 2
  • Submission successfully evaluated
  • Sub shows up in Dashboard
  • Sub shows up in Leaderboard for Challenge Question 1 - UW Dataset - version 2
• "Simulate" new UW Dataset version

More

• Identify how to support / organize different datasets for different questions (yet more than one questions could use the same dataset)
• Identify and set max runtime (on NCATS and UW)

• identify the information to show in the Results page
• how to deal with different versions of the dataset
• compile insights stated in #7

• data
• other challenges we know of (i’ve seen stuff through Kaggle, Devpost, MIT, and maybe others)

Deploy model-to-data configuration on UW site. Start from the configuration used for the EHR DREAM Challenge: Patient Mortality Prediction.

The Home page is drafted here: https://docs.google.com/document/d/1xs0bCEs2nvnd1I-7-wO4QCMS-7hkQiAtTY8louSQ1co/edit?ts=5e83859b#

Changes:

• Capitalization
• Remove underscore separators

**Description:** {description}
**Ranked features:** {ranked_feature}
**References:** {resources}

• Create GH repo with codebase of the baseline method
• Create README as a tutorial (see this example from the Metadata Automation DREAM Challenge)
• Make the repo publicly accessible

• fixes #64
• image now uses synapseclient 2.0

• Prepare email communication
  • Link to registration page (#1 )
  • Timeline
  • Challenge questions
• Send email

• Invite data contributors and explain to them what to expect and how to proceed
• Add to the challenge site

• Update agreement between Sage and UW
• Configure clickwrap

Code to reuse: https://github.com/DREAMChallenges/Celgene-Multiple-Myeloma-Challenge/blob/9fb4f7012b1614bfcce21654c52988c92f9dc335/performance_metrics/metrics.R#L8

set to 500

https://www.synapse.org/#!Synapse:syn21849256/wiki/601866

Create a video tutorial similar to the one that we have created for the Metadata Automation DREAM Challenge: https://www.youtube.com/watch?v=YTkocEZfOzs

Ask Verena is she can also make this tutorial

Tasks

• Make video tutorial
• Publish on newly created YouTube playlist under Sage channel

Depends on

• #11