zeno-ml / zeno Goto Github PK

Primarily:

Data processing/Model running with caching:

• Preprocessing
• Transformations
• Model inference

DataFrame operations:

• Data slicing

Users should be shown some automated "insights" about their slices/reports when they first land on the reports tab. This can include new failures, regressions, anomalies, etc. in tabular and graphical form.

• The filtering does not get reset when filter nodes are removed
• Lasso select filtering is not possible
• Metrics are not recomputed on filtered space from lasso

Some file formats you want to show less, others more.

For example, metric function might not have a mutation, not need the extra variables. Can check the signature and call the function with the right arguments.

For IMU data, sensor stuff

create a view for tabular data.

Maybe consider making a default table view for Zeno generally that shows the values for all columns? idk.

Users should be able to create folders to organize slices into.

show version and help

The least generalizable piece of Zeno is the instance view.

It would be great to make this modular, where people can install e.g. zeno-pointcloud and zeno then has an option to take in pointcloud data and visualize it without changing the zeno code itself.

e.g. if I want to create a slice for every category in a metadata column shouldn't have to do it one by one.

the Python API breaks down substantially when trying to use a multimodal model. Need to re-design and update the API to directly support multimodal models.

Slices don't sync being put in folder until clicked.

• On start takes minutes to initially start, then subsequent calls are fast (are parametric UMAP JIT compiling functions?)
• The results are not the best (modify the parametric UMAP or switch to ScVis or other)

For example, what if I want to find all images with dogs, and then filter for the dogs that are "red". The first slicing function has to indicate the bounding boxes the second slicing function should focus on.

Initial running of Zeno is super slow, even before running any computation. We should figure out why.

• When creating the polgyon on an intermediate node in the pipeline it errors out
• When creating one with the same name it breaks the frontend and backend

Have to manually run npm run build after every change when iterating, should set up an npm run dev

Since we re-render the whole sample view each time we change the table, preferences are re-written, such as showing the mask or just groundtruth for segmentation.

We should find a way to preserve these settings across re-renders of the sample view.

• rotating messes up the region labeler polygon
• Filtering to new spaces and reporjection is jarring (maybe tween the points atleast for reprojection?)

Push the new view packages to pypi and make available for new users

Currently you can only do single joins for filters, e.g. (A AND B OR C AND D)
We need to be able to make more complex filters such as (A AND C) OR !(D)

Generally keeping track of issues that will start to come up as we scale to larger datasets:

• Vega binning for histograms is quite slow, especially if done N times across all metadata
• Calculating metrics on the fly for every interaction/slice can be slow
• Requesting metrics using lists of IDXs can be slow if IDX lists go into millions
• Running projections, e.g. UMAP can be painfully slow, likely need approximations or smart caching.

Can't load in all the data at once, e.g. 100,000 images into memory. Have to batch it for both preprocessing and prediction. ALSO remove from slice API, everything should be metadata.

Instantiate Zeno, pass in functions, and run it - will open new page with UI.

This will require lots of thinking about how to structure the API - do we pass functions in? try to read the notebook itself?

Some metadata columns may be dates or text. We should have a cell visualization for them that provides some overview and filtering.

• text - regex filtering and examples results/count of results
• datetime - calendar selection

Support function types given the previous output

Including

• Matrix of transforms/slices
• [ ]

Either in exploration or analysis tab, let people create grid view that crosses slices like a confusion matrix to see metrics at different intersections.

Users should be able to select a group of instances and create a new slice. We should also support some views that help people expand their slices such as nearest neighbors and embeddings.

Users should be able to define tests given the mean and standard deviation of the model average instead of hard thresholds.

two main bugs:

• allow names to be changed -> requires deleting old slice from map and adding new one
• update metric when slice edited -> requires deleting result and requesting again

Some metrics should be run on every slice, provide an way to stipulate that

https://stackoverflow.com/questions/38834378/path-to-a-directory-as-argparse-argument

Create a thorough walkthrough for getting started with CIFAR explaining different concepts.