dm4ml / motion Goto Github PK

Users should be allowed to write tests for application logic

Whenever duplicating a record, the old record typically isn't written to. So there are null-valued records that exist. Figure out how to cleanly dispose of them.

Including cron-scheduled triggers

Right now, after fit is called, model outputs aren't regenerated. We need to do this.

See if fashion pipeline can be effectively rewritten with one schema instead of 2

There are 2 views I am thinking of:

• train-once
• run

For each view, describe the operations a user can run in the UI

--

Explore

Goal: enable developers to build the pipeline. Quick iteration on models, can slice and dice intermediates to inspect the data as they please. Success: time it takes to build a pipeline is less than half the time it takes without motion.

Key aspects: runs training ops only once using the minimum number of data points specified. Checkpoints inputs and outputs for every transform. allows user to run each transform, as well as whole pipeline.

3 types of cells:

• type: feature or label type creation
• transform: create a transform, set dependencies, can execute it
• free: forks/copies state and allows users to perform any read operations on intermediate state. this is kind of like a jupyter cell/free-for-all.

Test

Goal: simulate deployment on a batch of data. Understand performance as it would be, deployed. Success: when user deploys the pipeline to prod, they don't see a big performance drop soon after.

Key aspects: auto-runs retraining whenever models are getting stale. Users cannot add type or transform cells here. They simply run the pipeline on the ids they care about. They can also evaluate a function to measure performance.

2 types of cells:

• free: same as above
• evaluator: takes predictions and labels, and computes an evaluation metric. can either be a function or a class with state (to do incremental maintenance).

Try redis and polars

Currently we have a call like:

results = store.con.execute(
            "SELECT fashion.query.query, fashion.query.text_suggestion, fashion.catalog.permalink, fashion.catalog.img_url, fashion.query.img_score FROM fashion.query JOIN fashion.catalog ON fashion.query.img_id = fashion.catalog.id WHERE fashion.query.img_id IS NOT NULL AND fashion.query.query_id = ? ORDER BY fashion.query.img_score ASC",
            (query_id,),
        ).fetchdf()

We want to be able to support schema migrations---for example, a user adding a new key to a relation. Ideally the user does not have to run a separate command; Motion should automatically detect when a schema doesn't match up, and migrate to match up.

It would be good to prompt the user to confirm they want to migrate the schema.

Components:

• Header
• Pipeline
• Button
• Cell
• Imports
• Run

• Requirements.txt
• Schema documentation
• Store methods (common ones one will use)
• Transform class
• end to end example (including how to log feedback)

Don't need to make it official yet.

Triggers:

• scraper: scrapes recent headlines
• chatbot: puts recent headlines in prompt and pings LLM, along with user's interests

The example project is currently outdated.

Figure out how to take in new labels or feedback from the user. Also figure out how to create an evaluator.

• Basic get method
• Get method with guardrail (e.g., can't peek into future)
• Copy method (takes in id, doesn't execute triggers)

Helps with validation and removes the need for python 3.10

• Write components
• Write scrapers
• Write streamlit dashboard
• Write fine tuning

Use a generative image model like stable diffusion to:

• generate for a user's prompt

Needs fine-tuning on fashion images. We can use our catalog and fine-tune on text_suggestion, image pairs that the user likes.

Handle authentication and allow for nginx and/or gunicorn serving.

Right now the thread will stop if there's a failure in a cron thread. Make it fail gracefully.

Steps:

• User uploads camera roll + photo of themselves
• Model prunes set of images for images that have outfits of the user
• Find k most "worn" items for the user and store this
• Find similar clothing times to what users already have

Models:

• Face recognition to filter camera roll for pics of user
• Outfit segmentation to find most "worn" items
• Retrieval model to find neighboring outfits (scraped from online catalog) that the user doesn't own

First pass w/o fine-tuning

Default can be async

It looks like the on-disk version of duckdb is too slow. We may want to save the data directly in arrow tables and use duckdb to query. When a session shuts down, we need to:

• persist data/table information
• persist trigger state

• Rename component to transform
• Rename fit and transform methods to fit and infer
• Make a file for each transform

• Write basic documentation for transform lifecycle
• What should be async? Fit? How to coordinate?

Allow users to run triggers several times if a trigger fails. They can set this parameter in the constructor class?

In motion test, allow tests to run on clean data. We can essentially create a new version of the application.

This should just be an alias for executing a python script, checking if they are in the right directory.