aslotte / mlops.net Goto Github PK

We need some test automation to ensure that are properly setting values when creating experiments, runs and metrics

We would need to add integration tests to ensure that we can properly set up

• Experiments
• Runs
• Metrics
• Log models

The connection string could be kept as part of repo secrets

Some method names don't make it clear that they're async. For example RetrieveEntity method returns Task<TEntity>. To make it more clear that the method is async, consider renaming it RetrieveEntityAsync. Do the same for any other methods that return Task<T>.

Instead of passing in the connectionstring in the constructor, let's use a method, e.g.

public MLLifecycleManger UseAzureBlobStorage(string connectionString) 
{
     //do stuff here
     return this;
}

Add the ability to pass in the evaluation result from a regression model and automatically log all values as key-value pairs

Describe the bug
When calling CreateExperiment duplicate experiments are creating if one with the same name already exists in Azure TableStorage

To Reproduce
Steps to reproduce the behavior:

1. Call CreateExperiment two times in a row using Azure Storage

Expected behavior
I would expect that if an experiment with the same name already exist that no new experiment is created and rather the existing one's id is returned

Should contain

• Which issue it fixes (e.g. Fixes #1234)
• Description of change

• How to get started contributing
• Any coding style guidelines if we have that

We need the ability to get the best run in an experiement based on a specific metric.
For example, let's say we value the F1 score for a given machine learning model we are training (contained in an experiment). Each time we train the model (run), we want to know if this model is better than a previously recorded model or not.

The reason we want this is so we don't need to upload every model that we don't care about.

I'm thinking something like this

public void IsBestRun(Guid runId, string metricName) 
{
    // 1. Fetch the given metric value for that run
    // 2. Given the experiment that run is apart of, fetch the best run for that metric
    // 3. Compare if this is the best run or not
}

We should add /// <inheritdoc/> to all interface implementation methods/properties so they are properly registered in xml doc comments with the base documentation when they are referenced directly instead of as an interface.

Is your feature request related to a problem? Please describe.
Along with capturing the run and the corresponding metrics , we should capture Run duration as well.
Also is there a way to log hyperparameters used to train a model the same we log the metrics using a generic method? Is there a method or property on ml.net which returns a list of hyperparameters?

Describe the solution you'd like
Add 2 new properties for capturing run duration and hyperparameters on the Run class

Describe alternatives you've considered
N/A

Additional context
N/A

Is your feature request related to a problem? Please describe.
MLLifecycleManager is a mouthful. That's a problem.

Describe the solution you'd like
An easier term to understand and annunciate is ModelContext and IModelContext
As part of this issue, we would rename MLLifecycleManager to ModelContext

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

We should add an example solution on how this SDK can be used. It should probably go hand-in-hand with a page for documentation as well.

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
In order to provide good intellisense for our nuget package we need to add good XML documentation for our public properties and methods.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
Added an example of multi-class classification

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe.
The initial setup of MLLifeCycleManager for small test scenarios seems problematic in that you are required to set up a backing store before using or it throws an exception. It seems to me there should be a default implementation that requires zero configuration, perhaps an in-memory dictionary based store.

Describe the solution you'd like
Add in-memory store implementations of IMetaDataStore and IModelRepository - these will be useful for testing various scenarios in which the user does not need to persist the data in a permanent storage mechanism. Make these the default implementation instead of throwing an exception via EnsureStorageProviderConfigured on every call to MLLifeCycleManager

Describe alternatives you've considered
Alternatively could use SQLite implementation as default as it does not require any configuration by default, however it does persist on disk which might be undesirable for testing various scenarios.

Additional context
N/A

Add method to get all experiments from various meta data stores

Is your feature request related to a problem? Please describe.
Along with capturing the run and the corresponding metrics , we should capture training time as well. Also Refer to #76

Describe the solution you'd like
Add a new property to capture training time.

Describe alternatives you've considered
N/A

Additional context
N/A

Is your feature request related to a problem? Please describe.
I think it would be useful to know when one first started running an experiment.

Describe the solution you'd like
Add a new property for CreatedDate on IExperiment and it's associated implementations.
Note that this property should be immutable, e.g. when we run a new run for an experiment, this property should not be updated.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

The purpose of this ticket is to draw a very simple sketch on how the UI could look like for the web client.

I'm thinking the simplest possible.

We have the following layers

• Experiment, Runs, Metrics.

In the future we want to register models and deploy them but we can think of that UI later

Is your feature request related to a problem? Please describe.
Currently, we need two lines for creating a run, first creating an experiment then one for creating a new run.

Describe the solution you'd like
It would be nice to have a method like:

public Task<Guid> CreateRunAsync(string experimentName) 
{
}

What the method would do is create the experiment and then a run and return the run id.

Describe alternatives you've considered
N/A

Is your feature request related to a problem? Please describe.
While out running (that's when my ideas come) I thought of Sammy's question on how we would know which code was used to train a specific model. Given that we currently would run an e.g. a GitHub Action on a training run, or when running locally, have no GitHub commit at all, I can see that we easily would lose track of which run/artifact belonged to what code commit, e.g. model pipeline.

Describe the solution you'd like
A simple solution for us to start with is to add two columns on the run entity, both optional.

• GitHub commit hash
• Comments field

This would allow us to add the GitHub commit hash to the run, and if running locally a comment on what e.g. what the changes are that one is trying. Ideally we would then be able to link the web client with the repo in which the model was trained so that you could click on the run and it would automatically bring up the Git commit (e.g. PR) for which this code was trained on.

Describe alternatives you've considered
Open for suggestions.

Add the ability to pass in the evaluation result from a binary classifier and automatically log all values as key-value pairs.

Is your feature request related to a problem? Please describe.
Our current IsBestRun method tries to find the best run for a given metric by finding the largest value. This works well for 99% or so of cases but e.g. log loss entropy should be closer to zero to be a good value.

Describe the solution you'd like
Add ability to define what a good run looks like for a given metric

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Registering models and metadata in the cloud is not always feasible.
To that effect we want to be able to add a provider to store these things on-premise

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
Run all unit tests in the solution.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Create a simple Blazor App with a RESTful Web API backing it

Similarly to #3 we want the ability to upload models/artifacts to a local file share.

We can either make this the default case when using SQLite or we can create separate extension methods so a user can decide if they want to store this locally or not (e.g. you may want to mix Azure for meta data and local for models)

Describe the bug
SQLite is not making use of Data folder to store the db file.

To Reproduce
Steps to reproduce the behavior:

Expected behavior
Create the db in the Data folder

Add skeleton Blazor WebAssembly client that we can continue to build upon in smaller tickets.

We need to add

• License
• Version number

Anything else required to be in line with other packages.

Add the ability to pass in the evaluation result from a multi-class classifier and automatically log all values as key-value pairs.

During a run we want to upload a model to a container in Azure.

I'm envisioning the structure to look like a container name ModelRepository and each model being named the unique GUID for the run.

Add a method to get all runs for an experiment from various metadata stores

Add batch for the current build status

Background

We need to automate the build and deployment of our nuget packages.
Each package needs to have a consistent build number.

We should probably run the dotnet pack command with versioning after each CI build (we can have different versions for PR builds vs master builds).

The deployment to nuget.org should most likely be a separate workflow file as it should be manually triggered once we have a release.

Add method to get all metrics for a given run

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
Add GitHub workflow to publish packages from a release branch to nuget.org

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe.
To get started with the repo, we should provide instructions or scripts to install all dependencies needed to run unit/integration tests and build the project.

E.g.

• Setup SQLite (script or instructions)
• Setup Azure Storage Account (ARM Template)

@dcostea adding this issue so we remember to test it at some point :)