Use Case

As a user, I would like to see the wall clock time elapsed between frames or since the first frame. This is useful for understanding how fast in real time the cells are actually changing.

Acceptance Criteria

Currently the data is only represented by image number (aka frame number) in sequence.

1. We need to obtain the wall clock times from the data source and incorporate it in the data.
2. Displaying the time elapsed from start as an overlay on the view (or elsewhere) would complete this request.

Details

Bonus: It could also be interesting to display this time value in the plot of a track, to help understand how long a track is. This probably requires some nontrivial UX design. We could promote this bonus item to a new issue.

Use Case

It could be useful to have the track ID in the url query string so that users can send particular track urls to each other.

Acceptance Criteria

• When a cell (or track) is selected, update the url with ?dataset=X&track=Y
• also update url when current dataset is changed
• Initialize app from query string when dataset and track are provided.

Details

Bonus: add &t=time

Convert to a React architecture in order to make it easier to skin and re-lay-out the UI.

• #48
• #49
• #32
• #37
• #75
• #76
• #77
• #78
• #79
• #74
• #30

Description

When clicking on the background to deselect a cell, the highlight on the last selected cell persists. This goes away when the frame is changed, so likely the highlighted track is not being updated or cleared in ColorizeCanvas.

Expected Behavior

Highlight around last selected cell should be gone.

Use Case

The UI is becoming more heavily populated.
Soon we will have the main functionality complete (as requested and prioritized by scientists) and it is time to skin the app, give it some polish, and review the UX.
http://dev-aics-dtp-001.corp.alleninstitute.org/nucmorph-colorizer/dist/

Acceptance Criteria

At least a meeting with UX to initiate work and discuss goals.

Details

We hope that we don't need to rework too much of the UI but understand that a re-layout is in order. We want to keep this light and continue to be able to have quick turnaround to our scientists.

put a vertical marker at the current time in the plotly plot

Description

when changing dataset, the time slider did not reset to t=0 but the rest of the view did.

Expected Behavior

all ui in consistent state.

Reproduction

go to any nonzero time, and then change dataset.
observe time value different than main view.

Use Case

for a selected track, show the path from centroid to centroid up to the current time. this could be an overlay that draws a simple line list.

Use Case

Currently, when generate_data.py crashes, it keeps the frame data but loses all intermediate work (bounds.json). Because the data generation scripts usually take several hours to run, issues requiring a restart (unstable run environments, terminals closing, computers going to sleep) can delay dataset processing by a whole workday.*

Ideally, the generate_data.py would detect if frames are missing from the dataset and start from the first missing frame.

(*or more; I am currently going on three days for regenerating the full-resolution datasets 😢)

Acceptance Criteria

• generate_data.py makes backups of the bounds data to bounds.json.
• When making frames, the script will check if there is existing frame data in the directory and search for missing indices. If a missing frame is found, it will load the bounds.json and start writing bounds and frame data from that index.

Details

• Add a flag to force overwriting the existing dataset?

Use Case

Derek has data starting in a different format but convertable to this app

Acceptance Criteria

Write script to convert his data into this format and verify it can be loaded

Details

Get specs and input data from Derek

Use Case

For some data, playback is actually too fast. User Derek has requested making it adjustable.

Acceptance Criteria

UX

Use Case

Add data that has the xy position of the centroid, and the bounding box of every cell.

Acceptance Criteria

A click on the viewport can console.log the xy coordinates of the cell's centroid. Both bounding boxes and centroids in pixels.

Details

Can we load json data that contains a 2d array or array-of-pairs like this? Could also just store it as 1D array and then restructure it at load time.
Is it time to consider using a binary file format? -- is there a generic well known file format for numeric array data that can be written from python and read in JS that isn't totally custom? is it zarr?

Use Case

Internal-only, non-user affecting. Reduce code complexity by having TimeControls and RecordingControls' advancement loops use the same underlying logic, just with separate callback behavior.

Acceptance Criteria

• When recording, the interval loop should be completely contained in TimeControls.
• Any side-effects during recording should be done as callbacks.
• Wrapping or termination behavior should be handled via callbacks.

Details

e.g.:
private playTimeSeries(onNewFrameCallback: () => void, onCompleted: () => void, intervalMs: number): void {}

• Note: ok to have separate loops for timeout vs. interval if needed
• separate public entrypoint for recording

visualization: draw a small vector from current centroid to previous centroid (or next centroid) to indicate how the cell is moving.

Use Case

allow clicking to read back the id stored in the data, for cell id and track id

Users want to see the min and max feature value. For prototype, we could probably just print the numbers to the left and right of the displayed colormap

Use Case

I would like to eliminate certain feature values from the view

Acceptance Criteria

select a max and min value for a feature - could be interesting to do this as a two ended slider tht you can push in from either end.
I am assuming we would just hide cells that are outside the selected range. Requested by @niveditasa

Details

depends on #2
Where to put the controls? I imagine a slider just below the color gradient and lined up with it. Do we want to truncate the gradient or rescale the gradient to fit to the new min and max? If the latter, then lining it up with the slider would be misleading.

Use Case

Linting and type consistency is beneficial for code quality!

Acceptance Criteria

Add a gh action to automatically run lint and typecheck for this codebase. Also clean up all current lint and typecheck errors and warnings.

Details

Example action to copy in here:

https://github.com/allen-cell-animated/volume-viewer/blob/main/.github/workflows/ci.yml

Use Case

Requested by @jcass11 and @jessicasyu!
The requested feature is to indicate the existence of cells even if they're not currently being colored, giving them an alternate color instead of hiding them.

Acceptance Criteria

• Cells outside of the range should have an alternate color applied to them instead of color ramp values.
• Add some UI element to turn on/off this feature.

Details

Use Case

@jcass11 and @vianamp requested the ability to directly download MP4 video for use on presentations and sharing.

Acceptance Criteria

• Users can click a button to initiate a recording.
• Once the recording is finished, an encoded MP4 video will be downloaded to the user's Download directory (or the user will be prompted for a download location).
• Naming should default to the same as the existing RecordingControls.

Details

• Similar to simularium/simularium-viewer#252, which is running into technical issues with built-in browser security policies.

Use Case

Scientists are requesting that the features be renamed (currently NUC_shape_volume_lcc and NUC_position_depth) to more human-readable names (Depth or Volume?)

Acceptance Criteria

• Change feature names in the default dataset
• Future datasets generated by generate_data.py should also have matching feature aliases.

Details

This can be edited pretty easily in the existing datasets but this should also be done within the data generation script. (Currently the features are hardcoded: just add a string dictionary, perhaps?)

Add a button labeled "Load Datasets" that brings up a modal with a textbox. The modal will show two different text inputs, one for a collection URL (or JSON) or a URL for a single dataset.

Some button (Load) will attempt to load the data and close the modal. See Simularium for an example.

TODO: Data validation?

Use Case

Preparation for future UI overhaul!

• Convert existing HTML to JSX for controls
• Convert global variables into React state, separate logic and UI

Use Case

click on a cell, and allow a link to open a 3d viewer to show the cell at the given time, in 3d

Use Case

If we try to download images with zero delay, something goes wrong where files are skipped in a non reproducible way.
So we have set a hardcoded delay between frames.

Currently, the download speed for image frames is throttled at a set delay in RecordingControls.ts.

This value was set experimentally, but only works on the default datasets. Downloading frames from larger image resolutions (or on a slow computer) may lead to dropped frames.

Acceptance Criteria

• No files are dropped when downloading frame data on a higher resolution dataset (4x), tested across multiple computers?

Details

• ideally we can have no delay and not drop frames.
• we think we have to have a delay but the minimum delay is bandwidth dependent.
• Is there a way to determine download completion?
  • https://stackoverflow.com/questions/65934318/detect-and-notify-when-download-is-complete (??)
  • Javascript API, but only applies to browser extensions: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/downloads/DownloadItem
• A probably bad solution is scaling the delay with the size of the frame.
• This would also be resolved by #33.

Use Case

I would like to be able to enter a track id that I know about beforehand, and have the viewer display it.

Acceptance Criteria

Command button: "Go to track" --> open dialog to get number entry? --> advance time to beginning of track, plot track and highlight cell

Details

If you can't open a dialog then we should have a hide/show for the numeric input so it doesn't have to be kept up to date...?

Description

Change prefix.
Click start to save image sequence.
See that custom prefix resets to default and is not applied to saved images.

Expected Behavior

Custom file prefix should be used.

Use Case

As a data stakeholder, I have the ability to host my data on AWS and would like to use the tool to point to any cloud url.

Acceptance Criteria

nucmorph colorizer has a url query parameter for http://viewer_url/?dataset=dataset_url in which dataset_url points to the top level dataset json file. The remaining file paths referenced inside should be considered relative to this url.

Details

Take url string, chop off manifest.json and use the remaining string as baseUrl...

We will have to figure out what to do with the dataset dropdown box. That is TBD.

Ultimately we could have dataset=url&feature=featureName&track=trackId&t=time but this issue is only for dataset.

Use Case

Just to clean up the code a bit, we can encapsulate all the controls around locking min/max, reset, and setting min/max into a separate file.

Acceptance Criteria

main.ts gets smaller and simpler and forwards calls into some kind of Feature or FeatureController class.

Details

Does this really help anything?

Use Case

From user:

If I made the track overlays into gifs for each position could they be set as background channel for the plot instead of the gray? Would be great if background channel that is displayed was configurable so these plots could be used for validation of nuc and cell segmentation.
For nuc segmentation validation I would use the nuc channel alone. For CAAX seg validation I would use the nuc & caax channel as a 2 color RGB image.
Making it configurable would increase the use case tremendously

Acceptance Criteria

Prototype Idea: provide a set of background frames as pngs and composite the main view on top of them.

Details

The user may have multiple backdrop sets they want to try out. Probably need a blend-type parameter too.

Use Case

provide a time slider and numeric input

Description

Centroid coordinates are in the original dataset's coordinates, but don't match the downsampled pixel coordinates of the output frame (currently off by a factor of 4).

Expected Behavior

Centroid coordinates should be given in the pixel coordinates of the frame.

Description

Requested by @vianamp and @jcass11! Have the default color ramp match the upcoming publications, may be subject to change.

Use Case

Requested by @vianamp!
When comparing between datasets in a collection, the global min and max for a given feature should be shown. Currently, when switching between datasets, only the current feature min/max is used.

Acceptance Criteria

Details

• This could either be done at runtime (by fetching the feature data for each dataset in the collection) or when generating the collection metadata.
• Likely needs a UI feature (checkbox? Radio button?) to indicate behavior.

Use Case

show the actual time index as text

Add at least one diverging scheme with white in the middle
See:
https://matplotlib.org/stable/tutorials/colors/colormaps.html
scroll down to the "diverging" section for examples.

Use Case

As a quantitative scientific tool, we need to have units on numeric measured features.

Acceptance Criteria

Feature data should have units. This needs to come from the data sources.

In the viewer, we should see units on all places that display feature values.
In particular: the editable feature range values, the hovered feature value, and the plot Y axis. (I may have missed something here).

Use Case

show a plot below the viewport that plots the currently selected feature for the currently selected track

Use Case

Add a way to change the color of outliers/cells outside the current range.

Acceptance Criteria

• Dropdown or color selector for cell outlier colors.

Details

• See also #69.
• Should outliers (NaN) and values outside of range be two different categories?

Use Case

I want to produce a colorized movie to embed in powerpoint

Acceptance Criteria

"capture mode" button --> press Play or auto-play when capture is clicked --> disable UI (except for pause/cancel), iterate through sequence and save out pngs one by one --> exit capture mode when done or cancelled, re-enable UI. Name output files sequentially.

Details

• output resolution? canvas size or could we do original frame size?
• where to put the controls? right next to time slider?
• capture track could be interesting but was not asked for. do we start from t=0 or current t, start of track, etc?

Use Case

Even though the cell is colorized, as a user, I might want to know the actual feature value.

Acceptance Criteria

• Show the current feature value of the hovered cell in a box off the main display. If no cell is currently hovered over, the box should be empty.

Details

• Decide where to show output.

Use Case

Requested by Jessica and Graham!

They described an issue where, when doing a max-projection through a volume, cell nuclei (or cells) may be overlapping in the projection. Currently, there's no way to visualize the overlap in nucmorph-colorizer, and overlapping pixels must be assigned to one track or the other.

Jessica wants to use nucmorph colorizer to show spatial distributions of features, using intuitive colorizing tools.

Suggestions:

• Image overlay with different colorized segmentation data? (or brightfield/fluorescence image?)
• Multiple frames for different Z-stacks?
• Switching between different Z-slices represented as different datasets in a collection?

@jessicasyu @toloudis

Use Case

Requested by @jessicasyu!
The convention is for the darkest colors to indicate the highest values.

Acceptance Criteria

• Adjust all existing color ramps to match convention
• Optionally, add a way to invert color ramp choices.

Details

• Also, ideally update the URL with the color ramps and inverted state (&colormap=Sequential_Forest/inverted).

Use Case

If I have a dataset with a large number of features, I may not want to consume extra resources by loading them all initially.

Acceptance Criteria

Only load feature data on-demand.

Details

Could discard old feature data (or optionally keep around a very limited number of them in a local cache)

README hasn't been updated past default values and should be revisited.

Description

On the last frame in a track, cell tracks can still be selected but won't be highlighted on the canvas.

Expected Behavior

Cells should be highlighted for every frame that they're visible.

Reproduction

Skip to frame 569 and try clicking any cell.
http://dev-aics-dtp-001.corp.alleninstitute.org/nucmorph-colorizer/dist/index.html?dataset=Mama%20Bear&feature=NUC_shape_volume_lcc&track=8918&t=466
Go to the above link and advance the frame by one.

Use Case

Nucmorph is not the only project that can have data in this app. But currently the nucmorph datasets are hardcoded.

Acceptance Criteria

Populate the dataset dropdown box from (yet another) json file which will be the top-level "megaset" description

Details

URL query string could have megaset=M&dataset=D (probably we can come up with better word than megaset? how about "collection") or something like dataset=M/D perhaps?
Currently a dataset is a directory with a manifest.json inside.

For any feature, add some ui to allow the min and max of that feature to be locked in even when changing datasets.
Example:
load dataset1, feature1 (F1). F1.min and F1.max are determined and the colormap extremes are mapped to those values.
activate the "lock min/max" feature.
load dataset2. if we are still viewing F1 (?), then do not update the min/max values and keep the colormap range clamped to that min and max. values outside the range will be considered constant (clamped to the ends of the range) and will receive the colors at the ends of the range.

UI concerns: how to enable/disable this? when disabled, do things need to refresh or do we wait until some other user action? do we want to keep the same min/max even if it's a different feature with wildly different values? (changing feature would have to auto-disable this "lock min/max" thing).

Use Case

Outliers flicker on playback because they are given a unique color outside the active range. This is visually distracting when playing back animation sequence.

Acceptance Criteria

Checkbox: "Blend dropped data if < N Frames" where N can be user choice (some small number like 1 or 2 hopefully). Perhaps we could also give an option to have the cells disappear rather than get a special color.

Details

I think "blend" means to interpolate between two frames of data for cell on the same track. This would be a totally new implementation in the main shader to try to figure out how to color a given pixel.

Use Case

Show a scale bar on the canvas. Requested by @mogres!

Acceptance Criteria

TBD, may need help from UX in the future (@lynwilhelm)

Details

• May be similar to volume-viewer's solution
• Should be done in the dataset, during data generation.
• Relative to pixels or whole image?
• Overlay on top of viewport?