We missed a requirement: the metrics files should contain data sample counts for their relative operations. We can use any key names we like, so long as they're known at configuration time.

"Training samples", "Evaluation samples", "Validation samples", etc...

https://mlperf.github.io/mlcube/getting-started/hello-world/ isn't accessible

Cannot find the main function script and the docker file which were in the /build dir in the old repo.
Not sure whether it was removed on purpose, but I think it should be included.

This proposes a unified storage description for config 2.0.

Today, MLCube relies on a simple "file path" approach to describe the inputs and outputs of their tasks. However, for many platforms, such like Kubernetes, it is not possible to use a single file path, because they either have complex storage backend, or use their own layer of storage abstractions, which do not use "paths" to refer to the corresponding locations in the data storage. This proposal aims to address this problem by providing a unified way of describing storage that can cover both local file systems and more complex storage solutions.

A storage backend can be described in the platform section of the config, which is supplied by the user at run-time. The storage description consists of 2 main parts: a name that will be used as a reference in the tasks' I/O paths, and a platform-specific spec that provides the details of the storage backend in the target platform, so that the runner can use it to find the right location of data.
We do not change the "path"-like descriptions of task inputs/outputs in order to keep that simple, however, we do introduce a "variable"-like component as a part of the path, so that we can use this "variable" as a reference to the corresponding storage backend and use the rest of the path as a relative path to the given storage.
A most straight-forward example of such "variable" is "$WORKSPACE" which is currently being used to refer to a specific dir in local file system. With the new proposal, the "$WORKSPACE", or any "$CUSTOM_NAME" defined by user, can refer to an arbitrary storage backend as specified in the platform section.

Since the detailed spec of the storage is specified in the platform part, it can be decoupled from the shared MLCube config and only appear in the user's config. This also means that how the spec of a given storage backend is writtern should be agreed between a user and a runner, and not relevant to the MLCube publisher.
While we do not have to provide a standard for that specs, we may provide some "guidelines/examples" for popular platforms so that there can be a convention for runner implementors.

The following is an example of how the storage backend can be defined, notice the specs in the platform section and how they are used in the tasks section. Notice also that if we give the storage a name of "WORKSPACE" then we may redirect our default workspace to the specified storage backend, without change the values in the task I/Os.

name: example-mlcube
platform:
  storage:
  - name: K8S_DATA
    spec:
      kubernetes:
        pvc_name: my-pvc
  - name: NFS_DATA
    spec:
      nfs:
        host: 127.0.0.1
        port: 2049
        path: some/nfs/path
container:
  image: mlcommons/mnist:0.0.1
  build_context: "mnist"
  build_file: "Dockerfile"
tasks:
  download:
    io:
    - {name: data_dir, type: directory, io: output, default: $NFS_DATA/data}
    - {name: log_dir, type: directory, io: output, default: $NFS_DATA/logs}
  train:
    io:
    - {name: data_dir, type: directory, io: input, default: $K8S_DATA/data}
    - {name: parameters_file, type: file, io: input, default: $K8S_DATA/parameters/default.parameters.yaml}
    - {name: log_dir, type: directory, io: output, default: $K8S_DATA/logs}
    - {name: model_dir, type: directory, io: output, default: $K8S_DATA/model}

When I click on the MLCOmmans CLA I get the following:

The link redirects to https://github.com/mlcommons/systems/blob/main/mlcommons_cla.txt

THis makes it not possible for me to do pull requests. ALs I am already participating in MLcommons and have previously signed the CLA but under my google e-mail. Not sure if there is an issue with the process

MLCommons CLA bot:
Thank you for your submission, we really appreciate it. We ask that you sign our MLCommons CLA and be a member before we can accept your contribution. If you are interested in membership, please contact [email protected] .
0 out of 1 committers have signed the MLCommons CLA.
❌ @laszewsk
You can retrigger this bot by commenting recheck in this Pull Request

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

It would be nice to rename the mnist_openfl example to mnist_openflower to acknowledge how actively the Flower folks have been working with us.

Not a big deal, just trying to be kind to our friends!

It would be helpful to have aarch64 containers for the examples in addtition to x86_64, to be specific, hello_world is available in just x86_64