Segmentation/ Feature extraction has been redone by Erin in Cimini Lab.
Profile creation has been redone by Erin in Cimini Lab.
Data can be public in RODA Immediately

Update as generated:
Profile repo for image reprocessing: https://github.com/broadinstitute/cpg0017-rohban-pathways
Publication repo for initial image processing: https://github.com/carpenterlab/2017_rohban_elife
cpg0017-rohban-pathways

• Metadata completely filled out in Project Profiler Database (Imaging Platform internal use only)
• Segmentation/Feature extraction complete
• Profiling complete

Transfer to CellPainting Gallery:

• Upload data to RODA (is private by default)
• Run validation script to ensure completion
• Update cellpainting-gallery/README.md
• Make RODA entry public

If data is being published, prepare for publication:
- [ ] Run Distributed-BioFormats2Raw to create .ome.zarr files
- [ ] Upload (meta)data to IDR (images remain hosted in cellpainting-gallery).

Once published:
- [ ] Make IDR entry public

• Update cellpainting-gallery/README.md and open-data-registry/cellpainting-gallery.yml to reflect publication

We might end up moving https://github.com/carpenterlab/2016_bray_natprot/wiki to https://github.com/carpenterlab/2022_Cimini_NatureProtocols/wiki

If so, we'd need to update https://github.com/shntnu/open-data-registry/blob/b049ac9905205375343bbb74ce4dd2188df86a35/datasets/cellpainting-gallery.yml

See shntnu/open-data-registry#1 (comment)

Document the folder structure using these issues/notes as a starting point

• cytomining/profiling-handbook#54
• https://github.com/jump-cellpainting/aws/issues/53#issue-946391501
• https://github.com/jump-cellpainting/aws#steps-for-uploading-data-to-s3cellpainting-gallery
• https://github.com/jump-cellpainting/aws/blob/main/DATA_UPLOAD.md

Can cpg act as LFS for files generated in the publication of a dataset that are too large to be held in the publication repository itself?
If so, what structure would we want?

I propose that yes, we allow (but not require) cpg to host publication-associated large files that are too large to fit directly into a publication repo. When compared to the size of the rest of any dataset, publication-associated files are unlikely to cause any noticeable change in total size. For any publication, I would expect we would include download instructions for accessing the raw data in cpg anyway, so hosting publication-associated files has the major benefit of simplifying access to those files as well.

I suggest a folder structure of:

└── workspace
           └── publication_data
                      └── YEAR_FIRSTAUTHOR
                                  ├── large_file_example1.csv.gz
                                  ├── large_file_example2.csv.gz
                                  └── large_file_example3.csv.gz

@shntnu thoughts?

Requirement

Dear Community,

This is Robin from Novo Nordisk Pharmaceutical company.

Firstly, thanks to your contribution to project cellpainting-gallery.

As cellpainting-gallery is a public S3 bucket, We'd like to do some data analysis based the data you have on AWS S3.

I can list datasets with AWS CLI aws s3 ls --no-sign-request s3://cellpainting-gallery/cpg0000-jump-pilot/ .

seems it allows anonymous access the S3 bucket with AWS CLI , but when I access the S3 bucket via browser, it prompts me Access Denied.

http://s3.amazonaws.com/cellpainting-gallery/cpg0000-jump-pilot

<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>1Z4Z91ER4NJVXQYX</RequestId>
<HostId>0O1/TtCuJTVnxKUfT9tD4hfK78DTNAxLd/wEc4nYyGwhykyaKJzoZQljS+AKHjjYl1IbjRcJOIg=</HostId>
</Error>

We want to do that in a smart way, here we'd like to analyze data directly via s3 bucket instead of download datasets to local to analyze.

Because of limitation of local storage space, I'd like to know if it's possible to access the cellpainting-gallery's datasets via S3 REST API or S3 SDK for python (like boto3).

Let's take dataset cellpainting-gallery/cpg0000-jump-pilot as example, I'd like to get file size recursively through the whole dataset.

What prerequisites do I need to prepare for?

• Do I need to create an AWS Account?
• What default AWS S3 profile I need to prepare for accessing bucket cellpainting-gallery? like account and region.
• Would you please provide code snippet of S3 REST API or S3 SDK for python to retrieve the dataset?

It would be great if you could tell me the detailed steps to implement it?

Best Regards,
Robin

Data can be public in RODA Immediately
profile repo: https://github.com/broadinstitute/cmQTL
cpg0022-cmqtl

• Metadata collection form filled out by collaborator or Imaging Platform member (https://airtable.com/shrVxz9DcoMlDoCBI)
• Metadata completely filled out in Project Profiler Database (Imaging Platform internal use only)
• Segmentation/Feature extraction complete
• Profiling complete

Transfer to CellPainting Gallery:

• Upload data to RODA (is private by default)
• Run validation script to ensure completion
• Update cellpainting-gallery/README.md
• Make RODA entry public

If data is being published, prepare for publication:

• Run Distributed-BioFormats2Raw to create .ome.zarr files
• Upload (meta)data to IDR (images remain hosted in cellpainting-gallery).

Once published:

• Make IDR entry public
• Update cellpainting-gallery/README.md and open-data-registry/cellpainting-gallery.yml to reflect publication

Segmentation/ Feature extraction is being performed by Cimini lab (done)
Profile creation is being performed by Carpenter-Singh lab (done)
Data can be public in RODA Immediately

Update as generated:

Link to profile repo (same as publication repo): https://github.com/broadinstitute/profiling-resistance-mechanisms/tree/master/0.generate-profiles/profiles
Link to publication repo: https://github.com/broadinstitute/profiling-resistance-mechanisms
cellpainting-gallery identifier: cpg0028-kelley-resistance

• Metadata collection form filled out by collaborator or Imaging Platform member (https://airtable.com/shrVxz9DcoMlDoCBI)
• Metadata completely filled out in Project Profiler Database (Imaging Platform internal use only)
• Segmentation/Feature extraction complete
• Profiling complete

Transfer to CellPainting Gallery:

• Upload data to RODA (is private by default)
• Run validation script to ensure completion (https://github.com/jump-cellpainting/data-validation)
• Update cellpainting-gallery/README.md
• Make RODA entry public

If data is being published, prepare for publication:

• Run Distributed-BioFormats2Raw to create .ome.zarr files
• Upload (meta)data to IDR (images remain hosted in cellpainting-gallery).

Once published:

• Make IDR entry public
• Update cellpainting-gallery/README.md and open-data-registry/cellpainting-gallery.yml to reflect publication
• Move this Issue from cellpainting-gallery-private to cellpainting-gallery. This step can be performed at an earlier point if it needs inputs from an external collaborator.

Would be nice to point folks at a way to download.

Segmentation/ Feature extraction is being performed by Cimini lab (Erin)
Profile creation is being performed by Carpenter-Singh lab (Niranj)
Data can be public in RODA Immediately

Update as generated:
Link to profile repo: https://github.com/broadinstitute/2015_Bray_GigaScience-data
Link to publication repo: https://doi.org/10.1093/gigascience/giw014
cellpainting-gallery identifier: cpg0012-wawer-bioactivecompoundprofiling

• Metadata completely filled out in Project Profiler Database (entered as batch CDRP re-analysis in 2008 12 04 Imaging CDRP for MLPCN)
• Segmentation/Feature extraction complete
• Profiling complete

Transfer to CellPainting Gallery:

• Upload data to RODA (is private by default)
• Run validation script to ensure completion
• Update cellpainting-gallery/README.md
• Make RODA entry public

If data is being published, prepare for publication:
- [ ] Run Distributed-BioFormats2Raw to create .ome.zarr files
- [ ] Upload (meta)data to IDR (images remain hosted in cellpainting-gallery).

Data is already available at http://idr.openmicroscopy.org/webclient/?show=screen-1251

Once published:

• Make IDR entry public
• Update cellpainting-gallery/README.md and open-data-registry/cellpainting-gallery.yml to reflect publication
• Move this Issue from cellpainting-gallery-private to cellpainting-gallery. This step can be performed at an earlier point if it needs inputs from an external collaborator.

Note: This is a reprocessing of an existing dataset

Jamali et al should be listed as Jamali and Tromans-Coia et al; DOI is https://doi.org/10.1101/2023.02.15.528711

Looks like the second Chandrasekaran will be 2023 so can probably now not do 2022a, etc.

(Would do these myself but I don't have write access :( )

Hello, I would like to download the dataset cpg0000-jump-pilot to work on the 300+ compounds. My question is where to find the name of the compounds (is there a csv file for that) and is there an information about cell death modality induced by those 300+compounds (such as necrosis, apoptosis)? Many thanks in advance!

Hi there,

My name is Reese and I am a data scientist at ViQi Inc. working to do AI analysis on the jump datasets. I am currently looking at the jump pilot data and I had some questions regarding the JUMP-Target-1_compound_metadata_targets.tsv. What specifically do the genes in the target and target_list columns in this file represent? What source did it come from (pubchem, etc.)? Thank you so much for your time and continued efforts!

Cheers, Reese

[Link to publication repo]
cellpainting-gallery identifier = cpg0021-periscope

• Metadata collection form filled out by collaborator or Imaging Platform member (https://airtable.com/shrVxz9DcoMlDoCBI)
• Metadata completely filled out in Project Profiler Database (Imaging Platform internal use only)
• Segmentation/Feature extraction complete
• Profiling complete

Transfer to CellPainting Gallery:

• Upload data to RODA (is private by default)
• Run validation script to ensure completion (https://github.com/jump-cellpainting/data-validation)
• Update cellpainting-gallery/README.md
• Make RODA entry public

If data is being published, prepare for publication:

• Run Distributed-BioFormats2Raw to create .ome.zarr files
• Upload (meta)data to IDR (images remain hosted in cellpainting-gallery).

Once published:

• Make IDR entry public
• Update cellpainting-gallery/README.md and open-data-registry/cellpainting-gallery.yml to reflect publication
• Move this Issue from cellpainting-gallery-private to cellpainting-gallery. This step can be performed at an earlier point if it needs inputs from an external collaborator.

Formalize what kinds of questions should be asked here and make that info known.

• Make issue template for "question about dataset" that tells folks where to go instead (suggested add direct link for JUMP datasets and general description that applies to others)
• Add to README info about where to direct questions

Segmentation/ Feature extraction is being performed by (Cimini lab / Carpenter-Singh lab)
Profile creation is being performed by (Cimini lab / Carpenter-Singh lab)
Data can be public in RODA Immediately

Update as generated:
[Link to profile repo]
[Link to publication repo]
cellpainting-gallery identifier: cpg0025

• Metadata collection form filled out by collaborator or Imaging Platform member (https://airtable.com/shrVxz9DcoMlDoCBI)
• Metadata completely filled out in Project Profiler Database (Imaging Platform internal use only)
• Segmentation/Feature extraction complete
• Profiling complete

Transfer to CellPainting Gallery:

• Upload data to RODA (is private by default)
• Run validation script to ensure completion (https://github.com/jump-cellpainting/data-validation)
• Update cellpainting-gallery/README.md
• Make RODA entry public

If data is being published, prepare for publication:

• Run Distributed-BioFormats2Raw to create .ome.zarr files
• Upload (meta)data to IDR (images remain hosted in cellpainting-gallery).

Once published:

• Make IDR entry public
• Update cellpainting-gallery/README.md and open-data-registry/cellpainting-gallery.yml to reflect publication
• Move this Issue from cellpainting-gallery-private to cellpainting-gallery. This step can be performed at an earlier point if it needs inputs from an external collaborator.

Segmentation/ Feature extraction is being performed by Erin in Cimini lab
Profile creation is being performed by Erin in Cimini lab
Data can be public in RODA Immediately

Update as generated:
profiler repo = https://github.com/jump-cellpainting/cpg0014-JUMP-adipocyte
[Link to publication repo]
cellpainting-gallery identifier = cpg0014-jump-adipocyte

• Metadata collection form filled out by collaborator or Imaging Platform member (https://airtable.com/shrVxz9DcoMlDoCBI)
• Metadata completely filled out in Project Profiler Database (Imaging Platform internal use only)
• Segmentation/Feature extraction complete
• Profiling complete

Transfer to CellPainting Gallery:

• Upload data to RODA (is private by default)
• Run validation script to ensure completion (https://github.com/jump-cellpainting/data-validation)
• Update cellpainting-gallery/README.md
• Make RODA entry public

If data is being published, prepare for publication:

• Run Distributed-BioFormats2Raw to create .ome.zarr files
• Upload (meta)data to IDR (images remain hosted in cellpainting-gallery).

Once published:

• Make IDR entry public
• Update cellpainting-gallery/README.md and open-data-registry/cellpainting-gallery.yml to reflect publication

Segmentation/ Feature extraction has already been performed
Profile creation has already been performed
Data can be public in RODA Immediately

Update as generated:
[Link to profile repo]
[Link to publication repo]
cpg0018

• Metadata completely filled out in Project Profiler Database (Imaging Platform internal use only)
• Segmentation/Feature extraction complete
• Profiling complete

Transfer to CellPainting Gallery:

• Upload data to RODA (is private by default)
• Run validation script to ensure completion
• Update cellpainting-gallery/README.md
• Make RODA entry public

If data is being published, prepare for publication:

• Run Distributed-BioFormats2Raw to create .ome.zarr files
• Upload (meta)data to IDR (images remain hosted in cellpainting-gallery).

Once published:

• Make IDR entry public
• Update cellpainting-gallery/README.md and open-data-registry/cellpainting-gallery.yml to reflect publication
• Move this Issue from cellpainting-gallery-private to cellpainting-gallery. This step can be performed at an earlier point if it needs inputs from an external collaborator.

Segmentation/ Feature extraction is being performed by Cimini lab
Profile creation is being performed by Cimini lab
Data can be public in RODA Immediately - yes

Update as generated:
Link to profile repo
[Link to publication repo]
[cpg0020-varchamp-pilot]

• Metadata collection form filled out by collaborator or Imaging Platform member (https://airtable.com/shrVxz9DcoMlDoCBI)
• Metadata completely filled out in Project Profiler Database (Imaging Platform internal use only)
• Segmentation/Feature extraction complete
• Profiling complete

Transfer to CellPainting Gallery:

• Upload data to RODA (is private by default)
• Run validation script to ensure completion
• Update cellpainting-gallery/README.md
• Make RODA entry public

If data is being published, prepare for publication:

• Run Distributed-BioFormats2Raw to create .ome.zarr files
• Upload (meta)data to IDR (images remain hosted in cellpainting-gallery).

Once published:

• Make IDR entry public
• Update cellpainting-gallery/README.md and open-data-registry/cellpainting-gallery.yml to reflect publication
• Move this Issue from cellpainting-gallery-private to cellpainting-gallery. This step can be performed at an earlier point if it needs inputs from an external collaborator.

Segmentation/ Feature extraction is being performed by (Cimini lab / Carpenter-Singh lab)
Profile creation is being performed by (Cimini lab / Carpenter-Singh lab)
Data can be public in RODA Immediately

Update as generated:
[Link to profile repo]
[Link to publication repo]
[cellpainting-gallery identifier]

• Metadata collection form filled out by collaborator or Imaging Platform member (https://airtable.com/shrVxz9DcoMlDoCBI)
• Metadata completely filled out in Project Profiler Database (Imaging Platform internal use only)
• Segmentation/Feature extraction complete
• Profiling complete

Transfer to CellPainting Gallery:

• Upload data to RODA (is private by default)
• Run validation script to ensure completion (https://github.com/jump-cellpainting/data-validation)
• Update cellpainting-gallery/README.md
• Make RODA entry public

If data is being published, prepare for publication:

• Run Distributed-BioFormats2Raw to create .ome.zarr files
• Upload (meta)data to IDR (images remain hosted in cellpainting-gallery).

Once published:

• Make IDR entry public
• Update cellpainting-gallery/README.md and open-data-registry/cellpainting-gallery.yml to reflect publication
• Move this Issue from cellpainting-gallery-private to cellpainting-gallery. This step can be performed at an earlier point if it needs inputs from an external collaborator.

Segmentation/ Feature extraction is being performed by (Cimini lab / Carpenter-Singh lab)
Profile creation is being performed by (Cimini lab / Carpenter-Singh lab)
Data can be public in RODA Immediately

Update as generated:
[Link to profile repo]
https://doi.org/10.1101/2022.08.12.503783
cpg0019-moshkov-deepprofiler

• Metadata completely filled out in Project Profiler Database (Imaging Platform internal use only)
• Segmentation/Feature extraction complete
• Profiling complete

Transfer to CellPainting Gallery:

• Upload data to RODA (is private by default)
• Run validation script to ensure completion
• Update cellpainting-gallery/README.md
• Make RODA entry public

If data is being published, prepare for publication:
These are only training images (crops)

• Run Distributed-BioFormats2Raw to create .ome.zarr files
• Upload (meta)data to IDR (images remain hosted in cellpainting-gallery).

Once published:

• ~~ Make IDR entry public~~
• Update cellpainting-gallery/README.md and open-data-registry/cellpainting-gallery.yml to reflect publication
• Move this Issue from cellpainting-gallery-private to cellpainting-gallery. This step can be performed at an earlier point if it needs inputs from an external collaborator.

Segmentation/ Feature extraction is being performed by Cimini lab
Profile creation is being performed by (Cimini lab / Carpenter-Singh lab)
Data can be public in RODA Immediately

Update as generated:
[Link to profile repo]
[Link to publication repo]
cpg0026-lacoste_haghighi-rare-diseases

• Metadata collection form filled out by collaborator or Imaging Platform member (https://airtable.com/shrVxz9DcoMlDoCBI)
• Metadata completely filled out in Project Profiler Database (Imaging Platform internal use only)
• Segmentation/Feature extraction complete
• Profiling complete

Transfer to CellPainting Gallery:

• Upload data to RODA (is private by default)
• Run validation script to ensure completion (https://github.com/jump-cellpainting/data-validation)
• Update cellpainting-gallery/README.md
• Make RODA entry public

If data is being published, prepare for publication:

• Run Distributed-BioFormats2Raw to create .ome.zarr files
• Upload (meta)data to IDR (images remain hosted in cellpainting-gallery).

Once published:

• Make IDR entry public
• Update cellpainting-gallery/README.md and open-data-registry/cellpainting-gallery.yml to reflect publication
• Move this Issue from cellpainting-gallery-private to cellpainting-gallery. This step can be performed at an earlier point if it needs inputs from an external collaborator.

Segmentation/ Feature extraction is being performed by (Cimini lab / Carpenter-Singh lab)
Profile creation is being performed by (Cimini lab / Carpenter-Singh lab)
Data can be public in RODA Immediately

Update as generated:
[Link to profile repo]
[Link to publication repo]
[cellpainting-gallery identifier]

• Metadata collection form filled out by collaborator or Imaging Platform member (https://airtable.com/shrVxz9DcoMlDoCBI)
• Metadata completely filled out in Project Profiler Database (Imaging Platform internal use only)
• Segmentation/Feature extraction complete
• Profiling complete

Transfer to CellPainting Gallery:

• Upload data to RODA (is private by default)
• Run validation script to ensure completion (https://github.com/jump-cellpainting/data-validation)
• Update cellpainting-gallery/README.md
• Make RODA entry public

If data is being published, prepare for publication:

• Run Distributed-BioFormats2Raw to create .ome.zarr files
• Upload (meta)data to IDR (images remain hosted in cellpainting-gallery).

Once published:

• Make IDR entry public
• Update cellpainting-gallery/README.md and open-data-registry/cellpainting-gallery.yml to reflect publication
• Move this Issue from cellpainting-gallery-private to cellpainting-gallery. This step can be performed at an earlier point if it needs inputs from an external collaborator.

Hi
Thank you for your great work of Cell painting.
I want to download cpg0016-jump image data for analysis with meta data, howerver, I could not find metadata anymore(aws s3 ls --no-sign-request s3://cellpainting-gallery/cpg0016-jump/source_2/workspace/).

Can I get metadata for any other methods?

Bests

Segmentation/ Feature extraction is being performed by Cimini lab
Profile creation is being performed by Cimini lab
Data can be public in RODA Immediately

Update as generated:

Link to profile repo: https://github.com/broadinstitute/2019_08_03_Adipocyte_CellPainting_Claussnitzer
Link to publication: https://www.biorxiv.org/content/10.1101/2021.07.17.452050v1
cellpainting-gallery identifier: cpg0011-lipocyteprofiler

• Metadata completely filled out in Project Profiler Database (Imaging Platform internal use only)
• Segmentation/Feature extraction complete
• Profiling complete

Transfer to CellPainting Gallery:

• Upload data to RODA (is private by default)
• Run validation script to ensure completion
• Update cellpainting-gallery/README.md
• Make RODA entry public

If data is being published, prepare for publication:

• Run Distributed-BioFormats2Raw to create .ome.zarr files
• Upload (meta)data to IDR (images remain hosted in cellpainting-gallery).

Once published:

• Make IDR entry public
• Update cellpainting-gallery/README.md and open-data-registry/cellpainting-gallery.yml to reflect publication
• Move this Issue from cellpainting-gallery-private to cellpainting-gallery. This step can be performed at an earlier point if it needs inputs from an external collaborator.

Hesam's original top post is below

Hi Shantanu,
We are organizing our LipocyteProfiler data according to the outline. We populated the branches based on our available data, and I was wondering if the following information are sufficient for the cellpainting-gallery? if yes, how can we upload the data, and if no, what are the required information?

Thanks!

Segmentation/ Feature extraction is being performed by (Cimini lab / Carpenter-Singh lab)
Profile creation is being performed by (Cimini lab / Carpenter-Singh lab)
Data can be public in RODA Immediately

Update as generated:
[Link to profile repo]
[Link to publication repo]
cellpainting-gallery identifier: cpg0029-chroma-pilot

• Metadata collection form filled out by collaborator or Imaging Platform member (https://airtable.com/shrVxz9DcoMlDoCBI)
• Metadata completely filled out in Project Profiler Database (Imaging Platform internal use only)
• Segmentation/Feature extraction complete
• Profiling complete

Transfer to CellPainting Gallery:

• Upload data to RODA (is private by default) --> public in this case
• Run validation script to ensure completion (https://github.com/jump-cellpainting/data-validation)
• Update cellpainting-gallery/README.md
• Make RODA entry public --> will already be public

If data is being published, prepare for publication:

• Run Distributed-BioFormats2Raw to create .ome.zarr files
• Upload (meta)data to IDR (images remain hosted in cellpainting-gallery).

Once published:

• Make IDR entry public
• Update cellpainting-gallery/README.md and open-data-registry/cellpainting-gallery.yml to reflect publication
• Move this Issue from cellpainting-gallery-private to cellpainting-gallery. This step can be performed at an earlier point if it needs inputs from an external collaborator.

Would that be a good idea, to make the gallery as a whole easier to cite?

Would be nice to have approximate size of datasets (maybe list image and numerical data sizes separately) in the Available Datasets table in the README so folks wanting to use the datasets have some idea of what they are getting themselves into...