Code Monkey home page Code Monkey logo

covid-em-project's Introduction

MoBIE

MultiModal Big Image Data Sharing and Exploration

MoBIE is a framework for sharing and exploring large multi-modal image datasets. Example projects:

Examples videos

MoBIE software

For now, we provide two software tools to explore and generate MoBIE projects:

Tutorials

Tutorials for using MoBIE are available at http://mobie.github.io/, and installation instructions here

Data storage

To be accessible via MoBIE tools, your data needs to be organized as a project. One project can contain several datasets.

See the folder /data in this repository for an example project. It uses a small subset of the data from the PlatyBrowser.

Project layout

A project consists of a root folder with the file datasets.json that lists the available datasets and the datasets, each stored in a separate subfolder.

+datasets.json
+dataset1/
+dataset2/

Dataset layout

A dataset folder is structured as follows:

+images/
|  +--images.json
|  +--local/
|  +--remote/
+misc/
|  +--bookmarks
+tables/
+README.txt

with the following elements:

  • images/images.json lists the avaialable image data and stores their display options
  • images/local contains the metadata for image data stored locally, in bdv.xml data format
  • images/remote contains the metadata for image data stored remotely (s3 compatible cloud store) in bdv.xml data format
  • misc contains miscellaneous data
  • misc/bookmarks contains the bookmarks, stored as json files
  • tables contains tabular data associated with image segmentation data
  • README.txt gives a description of this dataset (optional)

images

MoBIE uses the BigDataViewer file format to represent image data, either stored locally or on a s3 object store:

The file images.json lists all image data that is available for a dataset. Currently, we support three different types of image data:

  • image: greyscale image data
  • segmentation: segmentaiton image data, where each segment is assigned a unique id; supports tables
  • mask: binary mask

misc/bookmarks

The bookmark folder must contain a file default.json that gives the bookmark for the initial view when opening the dataset. It may contain additional bookmarks stored as json.

tables

Tables are stored as tab separated values and can be read from the filesystem or git. For each segmentation with associated tables, the tables are stored in tables/<IMAGE-NAME>/. This folder must contain a table called default.csv, it can contain additional tables. All tables must contain the column label_id linking its rows to objects in the segmentation.

Contributors & Citation

MoBIE is primarily deveopled by Christian Tischer (@tischi), Kimberly Meechan (@K-Meech) and Constantin Pape (@constantinpape). It has been initially developed for the PlatyBrowser but has since grown in scope. In addition, Hernando Martinez Vergara (@HernandoMV), Martin Schorb (@martinschorb) and Valentyna Zinchenko (@vzinche) have contributed to the development.

If you use MoBIE for your research, please cite Whole-body integration of gene expression and single-cell morphology.

covid-em-project's People

Contributors

constantinpape avatar jhennies avatar martinschorb avatar tischi avatar

Stargazers

 avatar

Watchers

 avatar  avatar  avatar

covid-em-project's Issues

AWS credentials problem...

Hi,

when I load the remote data, I run into:

[ERROR] Module threw exception
java.lang.NullPointerException
	at bdv.img.n5.XmlIoN5S3ImageLoader.getAwsCredentialsProvider(XmlIoN5S3ImageLoader.java:93)
	at bdv.img.n5.XmlIoN5S3ImageLoader.fromXml(XmlIoN5S3ImageLoader.java:76)
	at bdv.img.n5.XmlIoN5S3ImageLoader.fromXml(XmlIoN5S3ImageLoader.java:49)
	at mpicbg.spim.data.generic.sequence.XmlIoAbstractSequenceDescription.fromXml(XmlIoAbstractSequenceDescription.java:111)
	at mpicbg.spim.data.generic.XmlIoAbstractSpimData.fromXml(XmlIoAbstractSpimData.java:153)
	at mpicbg.spim.data.generic.XmlIoAbstractSpimData.load(XmlIoAbstractSpimData.java:95)
	at de.embl.cba.bdv.utils.BdvUtils.openSpimData(BdvUtils.java:1356)
	at de.embl.cba.bdv.utils.sources.LazySpimSource.initSpimData(LazySpimSource.java:61)
	at de.embl.cba.bdv.utils.sources.LazySpimSource.wrappedVolatileSource(LazySpimSource.java:38)
	at de.embl.cba.bdv.utils.sources.LazySpimSource.getType(LazySpimSource.java:109)
	at de.embl.cba.bdv.utils.sources.LazySpimSource.getType(LazySpimSource.java:20)
	at bdv.util.BdvFunctions.addSource(BdvFunctions.java:572)
	at bdv.util.BdvFunctions.show(BdvFunctions.java:196)
	at de.embl.cba.mobie.viewer.SourcesPanel.showIntensitySource(SourcesPanel.java:387)
	at de.embl.cba.mobie.viewer.SourcesPanel.addSourceToViewer(SourcesPanel.java:362)
	at de.embl.cba.mobie.viewer.SourcesPanel.addSourceToPanelAndViewer(SourcesPanel.java:337)
	at de.embl.cba.mobie.viewer.SourcesPanel.addSourceToPanelAndViewer(SourcesPanel.java:294)
	at de.embl.cba.mobie.bookmark.BookmarksManager.addSourcesToPanelAndViewer(BookmarksManager.java:46)
	at de.embl.cba.mobie.bookmark.BookmarksManager.setView(BookmarksManager.java:32)
	at de.embl.cba.mobie.viewer.MoBIEViewer.<init>(MoBIEViewer.java:89)
	at de.embl.cba.mobie.viewer.MoBIEViewer.<init>(MoBIEViewer.java:49)
	at de.embl.cba.mobie.command.OpenMoBIEProjectCommand.run(OpenMoBIEProjectCommand.java:19)
	at org.scijava.command.CommandModule.run(CommandModule.java:199)
	at org.scijava.module.ModuleRunner.run(ModuleRunner.java:168)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:127)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:66)
	at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:228)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

This happens with both my personal as well as the read-only credentials.

bookmarks json file encoding

Hi,

it seems that the bookmark error Yannick was reporting (see below) is due to some weird character in the JSON file Covid19-S4-Area2/misc/bookmarks/bookmarks.json ":" in line 294.

I tried getting rid of it and did not manage.

Is there a good way to copy paste text from a google doc while not introducing these?
Maybe we should ask them to add bookmarks in some other text-only way...

Is there a java tool that you could use to sanitize this? Ana additional problem was the EOL conventions in the file (Windows vs. Unix).

Adding source: fibsem-raw...
Exception in thread "AWT-EventQueue-0" com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Unterminated object at line 294 column 2 path $.
	at com.google.gson.Gson.fromJson(Gson.java:942)
	at de.embl.cba.mobie.bookmark.BookmarksJsonParser.readBookmarksFromFile(BookmarksJsonParser.java:136)
	at de.embl.cba.mobie.bookmark.BookmarksJsonParser.parseBookmarks(BookmarksJsonParser.java:120)
	at de.embl.cba.mobie.bookmark.BookmarksJsonParser.readBookmarks(BookmarksJsonParser.java:105)
	at de.embl.cba.mobie.bookmark.BookmarksJsonParser.getBookmarks(BookmarksJsonParser.java:40)
	at de.embl.cba.mobie.viewer.MoBIEViewer.fetchBookmarks(MoBIEViewer.java:182)
	at de.embl.cba.mobie.viewer.MoBIEViewer.<init>(MoBIEViewer.java:77)
	at de.embl.cba.mobie.viewer.ActionPanel.switchDataset(ActionPanel.java:548)
	at de.embl.cba.mobie.viewer.ActionPanel.lambda$addDatasetSelectionUI$13(ActionPanel.java:536)
	at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022)
	at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2348)
	at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
	at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259)
	at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:252)
	at java.awt.Component.processMouseEvent(Component.java:6539)
	at javax.swing.JComponent.processMouseEvent(JComponent.java:3324)
	at java.awt.Component.processEvent(Component.java:6304)
	at java.awt.Container.processEvent(Container.java:2239)
	at java.awt.Component.dispatchEventImpl(Component.java:4889)
	at java.awt.Container.dispatchEventImpl(Container.java:2297)
	at java.awt.Component.dispatchEvent(Component.java:4711)
	at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4904)
	at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4535)
	at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4476)
	at java.awt.Container.dispatchEventImpl(Container.java:2283)
	at java.awt.Window.dispatchEventImpl(Window.java:2746)
	at java.awt.Component.dispatchEvent(Component.java:4711)
	at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:760)
	at java.awt.EventQueue.access$500(EventQueue.java:97)
	at java.awt.EventQueue$3.run(EventQueue.java:709)
	at java.awt.EventQueue$3.run(EventQueue.java:703)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74)
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:84)
	at java.awt.EventQueue$4.run(EventQueue.java:733)
	at java.awt.EventQueue$4.run(EventQueue.java:731)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74)
	at java.awt.EventQueue.dispatchEvent(EventQueue.java:730)
	at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:205)
	at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
	at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
	at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)
Caused by: com.google.gson.stream.MalformedJsonException: Unterminated object at line 294 column 2 path $.
	at com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1568)
	at com.google.gson.stream.JsonReader.doPeek(JsonReader.java:491)
	at com.google.gson.stream.JsonReader.hasNext(JsonReader.java:414)
	at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:184)
	at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:145)
	at com.google.gson.Gson.fromJson(Gson.java:927)
	... 44 more

permissions

Hi,

I am trying to push a new branch to add new data and fix the local repo.

Password for 'https://[email protected]': 
remote: Permission to mobie-org/covid-em-datasets.git denied to martinschorb.
fatal: unable to access 'https://github.com/mobie-org/covid-em-datasets/': The requested URL returned error: 403

conversion crash

Hi @constantinpape ,

at the very last step I got this:

I guess this is due to some changes in the metadata formats...

This progress looks :) because there were no failed tasks or missing dependencies

===== Luigi Execution Summary =====

In order to add the data to the EMBL S3, please run the following command:
mc cp -r data/Covid19-S5-mock-Cell1-2/images/local/fibsem-raw.n5/ embl/covid-fib-sem/Covid19-S5-mock-Cell1-2/images/local/fibsem-raw.n5/
Traceback (most recent call last):
  File "/g/emcf/common/5792_Sars-Cov-2/covid-em/stack_to_mmb_format.py", line 90, in <module>
    args.target, args.max_jobs, args.time_limit)
  File "/g/emcf/common/5792_Sars-Cov-2/covid-em/stack_to_mmb_format.py", line 58, in stack_to_mmb
    add_dataset(dataset_name, ROOT)
  File "/g/emcf/common/5792_Sars-Cov-2/covid-em/mmb_utils/metadata.py", line 53, in add_dataset
    json.dump(datasets, f, sort_keys=True, indet=2)
  File "/g/emcf/common/5792_Sars-Cov-2/covid-em/software/miniconda3/envs/mmb-dev/lib/python3.7/json/__init__.py", line 176, in dump
    default=default, sort_keys=sort_keys, **kw).iterencode(obj)
TypeError: __init__() got an unexpected keyword argument 'indet'

[1]+  Exit 1  

Accessability without aws config

Hey @constantinpape,

Our collaborator was trying to access the FIB-SEM datasets and failed, while he was able to access the tomo data.
I could reproduce by removing the aws credentials from my home, then the tomo stuff works but the FIB doesn't.

I guess, we also want the FIB data public, do you know how to fix this?

Cheers,
Julian

bookmarks

Hi @tischi,

I put some bookmarks locally but I get:

[ERROR] Module threw exception
java.lang.NullPointerException
	at de.embl.cba.mobie.bookmark.BookmarksManager.setView(BookmarksManager.java:29)
	at de.embl.cba.mobie.viewer.MoBIEViewer.<init>(MoBIEViewer.java:89)
	at de.embl.cba.mobie.viewer.MoBIEViewer.<init>(MoBIEViewer.java:49)
	at de.embl.cba.mobie.command.OpenMoBIEProjectAdvancedCommand.run(OpenMoBIEProjectAdvancedCommand.java:22)
	at org.scijava.command.CommandModule.run(CommandModule.java:199)
	at org.scijava.module.ModuleRunner.run(ModuleRunner.java:168)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:127)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:66)
	at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:228)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

What is wrong with the file? I cannot spot the problem...

data\Covid19-S4-Area1\misc\bookmarks

add segmentations

Hi,

Julian @jhennies has the segmentations almost ready.
What would be the workflow to add those? I saw that it is basically additional datasets (xml). But how does MoBIE know that it is a segmentation?

Local data does not load

Hi,

I can't load local data:

(Fiji Is Just) ImageJ 2.0.0-rc-69/1.52p; Java 1.8.0_202 [64-bit]; Mac OS X 10.14.6; 182MB of 3461MB (5%)
 
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.UnsupportedOperationException: Could not open or parse versions file: /Volumes/emcf/common/5792_Sars-Cov-2/covid-em/data
	at net.imagej.legacy.LegacyService.runLegacyCompatibleCommand(LegacyService.java:307)
	at net.imagej.legacy.DefaultLegacyHooks.interceptRunPlugIn(DefaultLegacyHooks.java:166)
	at ij.IJ.runPlugIn(IJ.java)
	at ij.Executer.runCommand(Executer.java:137)
	at ij.Executer.run(Executer.java:66)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.lang.UnsupportedOperationException: Could not open or parse versions file: /Volumes/emcf/common/5792_Sars-Cov-2/covid-em/data
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at net.imagej.legacy.LegacyService.runLegacyCompatibleCommand(LegacyService.java:303)
	... 5 more
Caused by: java.lang.UnsupportedOperationException: Could not open or parse versions file: /Volumes/emcf/common/5792_Sars-Cov-2/covid-em/data
	at de.embl.cba.mobie.dataset.DatasetsParser.datasetsFromDataSource(DatasetsParser.java:29)
	at de.embl.cba.mobie.viewer.MoBIEViewer.<init>(MoBIEViewer.java:65)
	at de.embl.cba.mobie.viewer.MoBIEViewer.<init>(MoBIEViewer.java:49)
	at de.embl.cba.mobie.command.OpenMoBIEProjectCommand.run(OpenMoBIEProjectCommand.java:19)
	at org.scijava.command.CommandModule.run(CommandModule.java:199)
	at org.scijava.module.ModuleRunner.run(ModuleRunner.java:168)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:127)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:66)
	at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:228)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.