📫 Feedback on usage issues: [email protected] / Professional consulting for server equipment: [email protected]

Label Sudio is also an external labeling system supported by YMIR and can be installed as an alternative labeling tool.

1. In the YMIR directory, modification Env file, configure label studio port：

LABEL_TOOL=label_studio
# Note format: LABEL_TOOL_HOST_IP=http(s)://(ip)
LABEL_TOOL_HOST_IP=set_your_label_tool_HOST_IP
LABEL_TOOL_HOST_PORT=set_your_label_tool_HOST_PORT

2. After configuring the label tool (LABEL_TOOL), IP (LABEL_TOOL_HOST_IP), and port (LABEL_TOOL_HOST_PORT) start the installation of label studio command:

docker-compose -f docker-compose.label_studio.yml up -d

It is recommended not to use the sudo command, as it may result in insufficient privileges.

3. Check the status of label studio:

docker-compose -f docker-compose.label_studio.yml ps

The user can access label studio through the default URL http://localhost:12007/. The installation is successful if the login page shows up.

4. Configure the Label Studio authorization token

  After the user registers and log in to Label Studio, select "Account & Settings" in the upper right corner of the page. Then, copy the token value and paste it into the corresponding location (LABEL_STUDIO_TOKEN) in the .env configuration file of the YMIR project. An example is as follows:

label studio env

LABEL_TOOL_URL=http://(ip):(LABEL_TOOL_PORT)

LABEL_TOOL_PORT=set_your_label_tool_port

LABEL_TOOL_TOKEN="Token token_value"

LABEL_TASK_LOOP_SECONDS=60

Restart ymir after configuring the token (LABEL_STUDIO_TOKEN).

5. The command to stop the label studio service is:

docker-compose -f docker-compose.label_studio.yml down

This section uses a complete model iteration process as an example to illustrate how to use the YMIR platform.

As shown in the figure above, YMIR divides the model development process into multiple steps. Details about how to run each step are listed in the subsequent sections.

Data and labels are necessary for the training of deep learning, and the training requires a large amount of data with labels. However, what exists in reality is a large amount of unlabeled data, which is too costly in terms of labor and time if all of them are manually labeled.

Therefore, YMIR platform, through active learning, first attains an initial model by local import or a small amount of labeled data, and uses this initial model to mine the most beneficial data for model capability improvement from a large amount of data. After the mining is completed, only this part of the data is labeled and the original training dataset is expanded efficiently.

The updated dataset is used to train the model again to improve the model capability. The YMIR platform provides a more efficient approach than labeling the entire data and then training it, reducing the cost of labeling low-quality data. Through the cycle of mining, labeling, and training, high quality data is expanded and the model capability is improved.

When you need to import a dataset with annotation files, please make sure the annotation type belongs to the existing label list of the system, otherwise you need to go to the label management interface to add custom labels in order to import the data. The following figure shows:

This time we add the tags 'helmet_head' 'no_helmet_head' to the list, the primary name and alias of the label indicate the same type of label. When the annotation of some dataset contains alias, it will be merged to primary name when importing. For example, if the label list contains the 'bike' (alias 'bicycle'), and a dataset A (containing only the 'bicycle') is imported, it will be displayed as 'bike' in the dataset details after import.

Users create projects according to their training goals(helmet_head，no_helmet_head) and set the target information such as mAP value, iteration rounds, etc. of the goals. As shown in the figure below：

The user prepares the mining set to be used for data mining (which may not need to contain annotation files) and datasets with training targets (training set and test set) for training an initial model. Before importing, please ensure that the format of the dataset meets the following requirements:

• The dataset is in.zip format, it should contain two folders named as "images" and "annotations" respectively;
• Images: create an ''images'' folder and place images in it. The formats currently supported by this platform are limited to jpg, jpeg, and png;
• Annotations: create an "annotations" folder and place annotation files formatted as Pascal VOC (if there are no annotation files, leave the folder empty);

The platform supports four kinds of dataset importing: public dataset replication, network importing, local importing, and path importing.

(1) public dataset replication: the user can copy the built-in dataset of the super administrator to the current operating user. The user can filter and import the label categories they need, as shown in the figure below:

Select the dataset and choose whether you want to synchronize the labels contained in the public dataset, click [OK] to start copying.

(2) Network import: users need to enter the URL path corresponding to the dataset as shown in the following:

(3) Local import: users needs to upload a zip file of the local dataset in the following format. The zip size is recommended to be no more than 200MB.

Users can download the example Sample.zip for reference as follows:

(4) Path Import:

1. Download the open-source dataset VOC2012 (Click to download VOC2012) and unzip it. Change the folder name as required, and then compressing them separately into zip packages that meet the import requirements.

2. Place dataset VOC2012 under ymir-workplace/ymir-sharing.

3. Select 'path import' and enter the absolute path of the dataset in the server: voc2012, as shown in the figure below:

After finishing the import of the initial dataset, click [Dataset Setting] to complete the corresponding dataset and mining strategy settings. The training set has been set as the default system training set when creating the project, and cannot be changed.

The user prepares the model for the initial iteration, either by local import or by model training. For local import, it is necessary to ensure that the model is in the required format.

• Only models generated by the YMIR system are supported.
• The uploaded file should be less than 1024 MB.
• the detection target of the uploaded model file should be consistent with the project target.

Model training can be done by clicking the [Training] operation button on the dataset list interface to jump to the Create Model Training interface, as shown in the following figure：

Select the training set (train1 V1), select the test set (val V1), select the training target (helmet_head, no_helmet_head), select the pre-training model (not required), training docker, training type, algorithm framework, backbone network structure, number of GPUs and configure the training parameters (training parameters provide default values, the default parameters in the key value can not be modified, the value value can be modified, if you want to add parameters can be added), click create task. If you want to add parameters, you can add them yourself), click Create Task. As shown in the figure below, the initial model is trained.

After successful creation, users can view the corresponding task progress and information on the task management page. Users can view the accuracy of the trained model (mAP value) after the task is completed.

After finishing the import or training of the initial model, click [Select Model] to select the initial model and finish the setup.After both the iteration data and the model are prepared, the iteration is started.

• Model iterations (Improve accuracy through iteration)

When iteration is enabled, YMIR provides a standardized model iteration process and helps users fill in the results of the previous operation by default in each step of the operation, so that ordinary users can follow the established steps to complete the complete model iteration process.

YMIR provides data mining algorithms that support million-level data mining to quickly find the most favorite data for model optimization.

[Mining Data Preparation] provides users with the data to be mined, and the original data set here is the mining set set set by project setting by default. The operation process is shown in the following figure.

Click [Next] after the operation is completed to open the [Data Mining] process.

The user can use the model obtained from the initial training to perform data mining on the dataset to be mined. Click the [Data Mining] button to jump to the data mining interface, as shown in the following figure.

The default original dataset is the result dataset prepared from the last mining data, and the default model is the initial model set in the iterative preparation. The user must also enter the filter TOPK as 500 (the first 500 successfully mined images), and set custom parameters if necessary.

After successful creation, users can view the mining progress and the result on the dataset management page.

If the data mined in the previous step does not have labels, users need to label them. Users can click the [Label] button on the task management page to jump to the data annotation interface as shown in the following figure.

The default original dataset is the result dataset obtained from the last mining. The user must also ente rthe email address of the annotator, and the labeling target (helmet_head, no_helmet_head). If you want to check the labeling platform by yourself, please click "View on labeling platform" and fill in your labeling platform account. If you have more detailed requirements for the annotation, you can upload the annotation description document for the annotator's reference. You must register with the labeling system in advance. You can click "Register Labeling Platform Account" at the bottom to jump to the Label Studio labeling platform to register their labeling account. Click on Create Task, as shown in the figure below:

After successful creation, users can view the labeling progress and other information on the dataset management interface. After the operation completed, the YMIR will automatically retrieve the annotation results and generate a new dataset with the new annotation.

After the labeling is completed, the labeled data set is merged into the training set and the merged results are generated into a new version of the training set. The following figure shows.

After the merging is completed, the model is trained again to generate a new version of the model, as shown below.

Users can download the models that meet their expectations. Or continue to the next iteration to further optimize the model.

After training the model, users can validate the model. On the [Model Management] page, you can click the [Verify] button of the corresponding model to jump to the [Model Validation] page. As shown in the following figure:

Select the validation mirror, adjust the parameters, click the [Upload Image] button, select the local image to upload, click [Model Validation], and display the results as follows.

Users can click the [Download] button on the [Model List] page. The downloaded file is a tar package, which contains the network structure of the model, network weights, hyper-parameter configuration files, training environment parameters, and results. As shown below:

This chapter contains the instructions for the YMIR-CMD. If you need to use the GUI, please refer to Ymir-GUI Installation.

Mode I. Pip Installation

# Requires >= Python3.8.10
$ pip install ymir-cmd
$ mir --vesion

Mode II. Installation from the source

$ git clone --recursive [email protected]:IndustryEssentials/ymir.git
$ cd ymir/command
$ python setup.py clean --all install
$ mir --version

The above figure shows a typical process of model training: 1) the user prepares external data, 2) imports it into the system, 3) appropriately filters the data, and 4) begins training to obtain a model (possibly with low accuracy). 5) selects images in a dataset to be mined that are suitable for further training based on this model, 6) annotates these images, 7) merges the annotated results with the original training set, and 8) uses the merged results to run the training process again to obtain a better model. This section implement the process shown above using the command line. All the following commands are prefixed with $ (which is also the Linux prompt under the normal user). When entering a command in the console, the user does not need to enter $ at the same time.

The system has the following requirements for external data.

1. With VOC annotations

2. The paths to all images (collectively referred to as assets or media in this system) need to be written uniformly in the "index.tsv" file. All annotation files need to be in the same directory.

3. The user must have read access to "index.tsv," all images, and all annotation files.

We take the pascal 2017 test dataset as an example. Download the dataset "VOC2012test.tar" from the official website and unpack.

$ tar -xvf VOC2012test.tar

After unpacking, the following directory structure is available (assuming VOCdevkit is in the /data directory).

/data/VOCdevkit
` - VOC2012
    |-- Annotations
    |-- ImageSets
    |-- Action
    |-- Layout
    |-- Main
    | `-- Segmentation
    ` -- JPEGImages

Note that all annotations are in the annotations directory and all images are located in the JPEGImages directory.

Users can use the following command to generate the "index.tsv" file.

$ find /data/VOCdevkit/VOC2012/JPEGImages -type f > index.tsv

The generated "index.tsv" is as follows:

/data/VOCdevkit/VOC2012/JPEGImages/2009_001200.jpg
/data/VOCdevkit/VOC2012/JPEGImages/2009_004006.jpg
/data/VOCdevkit/VOC2012/JPEGImages/2008_006022.jpg
/data/VOCdevkit/VOC2012/JPEGImages/2008_006931.jpg
/data/VOCdevkit/VOC2012/JPEGImages/2009_003016.jpg
...

And this "index.tsv" can be used for the next step of data import.

In addition, each annotation in the Annotations folder has the same main file name as the image. One of the xxx attributes will be extracted as a predefined keyword to be used in a later step of data filtering.

The command line on this system uses a method similar to Git to manage user resources. Users create their own mir repository and perform all the following tasks in this mir repo.

Use the following command to create a mir repo:

$ mkdir ~/mir-demo-repo && cd ~/mir-demo-repo # Create the directory and enter
$ mir init # init this directory to a mir repo
$ mkdir ~/ymir-assets ~/ymir-models # Creates assets and models storage directory, mir repo only keeps reference to assets and models

All type labels in mir repo are managed by labels.yaml. Open file ~/mir-demo-repo/.mir/labels.yaml, and you can see the following contents:

labels:
- create_time: 1646728410.570311
  id: 0
  update_time: 1646728410.570311
  name: frisbee
- create_time: 1646728410.570311
  id: 1
  update_time: 1646728410.570311
  name: car

You can add your own class ids and names, just like:

labels:
- create_time: 1646728410.570311
  id: 0
  update_time: 1646728410.570311
  name: frisbee
- create_time: 1646728410.570311
  id: 1
  update_time: 1646728410.570311
  name: car
- create_time: 1646728410.570311
  id: 2
  update_time: 1646728410.570311
  name: tv

There could be one or more alias for each type label, for example: if television and tv_monitor sepecified as the alias for tv, the labels.yaml could be changed to:

- create_time: 1646728410.570311
  id: 0
  update_time: 1646728410.570311
  name: frisbee
- create_time: 1646728410.570311
  id: 1
  update_time: 1646728410.570311
  name: car
- create_time: 1646728410.570311
  id: 2
  update_time: 1646728410.570311
  name: tv
  aliases:
  - television
  - tv_monitor

You can edit this file by vi and other text editing tools. You can add alias to type labels or add new type labels, but it is not recommended to change or remove the id and name of any type label that already exists.

The file labels.yaml can be shared among multiple mir repos by establishing soft links.

Users are required to prepare three data sets in advance.

1. A training set (for example, named "dataset-training"), with annotations, for initial model training.

2. A validation set (named "dataset-val") that includes annotations.

3. Mining set (named "dataset-mining"): a large dataset to be mined from.

The user imports the three data sets with the following command:

$ cd ~/mir-demo-repo
$ mir import --index-file /path/to/training-dataset-index.tsv \ # Path to dataset index
             --gt-dir /path/to/training-dataset-annotation-dir \ # Path to annotation dir
             --gen-dir ~/ymir-assets \ # Path to ymir-assets
             --unknown-types-strategy stop \ # Choose from stop, ignore, add
             --anno-type det-box \ # Choose from det-box, seg-poly, seg-mask
             --dst-rev 'dataset-training@import' # branch and task name
$ mir checkout master
$ mir import --index-file /path/to/val-dataset-index.tsv \
             --gt-dir /path/to/val-dataset-annotation-dir \
             --gen-dir ~/ymir-assets \
             --unknown-types-strategy stop \
             --anno-type det-box \
             --dst-rev 'dataset-val@import'
$ mir checkout master
$ mir import --index-file /path/to/mining-dataset-index.tsv \
             --gt-dir /path/to/mining-dataset-annotation-dir \
             --gen-dir ~/ymir-assets \
             --unknown-types-strategy stop \
             --anno-type det-box \
             --dst-rev 'dataset-mining@import'

• Note: Set optional --gt-dir to ground truth directory, and optional --pred-dir to prediction directory to import prediction and ground truth to ONE dataset.

Users can use this command to see the branches of the current mir repo:

$ git branch

This repo has four branches: master, dataset-training, dataset-val, and dataset-mining. The current repo is on the branch dataset-mining.

Users can also view the status of branch with:

$ mir show --src-rev dataset-mining

Users can merge dataset-training and dataset-val with:

$ mir merge --src-revs tr:dataset-training@import;va:dataset-val@import \ # source branches to be merged
            --dst-rev tr-va@merged \ # destination branch and task name
            -s host # conflicts resolve strategy: use infos on host branch (the first branch in --src-revs)

If the dataset-training and dataset-val before merging have 2000 and 1510 images, you can see that the merge branch has 2000 images as the training set and 1510 images as the validation set.

If the user only wants to train the model to detect humans and cats, we first filter out the resources of humans and cats from the tr-va branch:

$ mir filter --src-revs tr-va@merged \
             --dst-rev tr-va@filtered \
             -p 'person;cat'

First, users need to pull the training and mining docker images from the docker hub as follows:

$ docker pull youdaoyzbx/ymir-executor:ymir1.3.0-yolov5-cu111-tmi

and start the training process with the following command:

mir train -w /tmp/ymir/training/train-0 \
          --media-location ~/ymir-assets \ # assets storage dir
          --model-location ~/ymir-models \ # model storage dir
          --task-config-file ~/training-config.yaml \ # training config file, get it from training docker image
          --src-revs tr-va@filtered \
          --dst-rev training-0@trained \
          --executor youdaoyzbx/ymir-executor:ymir1.3.0-yolov5-cu111-tmi # docker image name

After the model training is completed, the system will output the model ID. The user can view the package file of the model in "/ymir-models".

This model is trained on a small dataset, and users can get the best images for the next training step in this mining process with the following command:

$ mir mining --src-revs dataset-mining@import \ # mining dataset branch
             --dst-rev mining-0@mining \ # destination branch
             -w /tmp/ymir/mining/mining-0 \ # tmp working dir for this task
             --topk 200 \ # topk
             --model-location ~/ymir-models \
             --media-location ~/ymir-assets \
             --model-hash <hash> \ # model id
             --asset-cache-dir /tmp/ymir/cache \ # asset cache
             --task-config-file ~/mining-config.yaml \ # mining config file, get it from mining docker image
             --executor youdaoyzbx/ymir-executor:ymir1.3.0-yolov5-cu111-tmi # mining docker image name

• Note: Argument --add-prediction will add prediction boxes to result dataset.

Now the user has 200 images on the branch "mining-0". These images will be most useful in the next training step. The next task is to export these resources and send them to the annotators for labeling.

Users can export assets with the following command:

$ mir export --asset-dir /tmp/ymir/export/export-0/assets \ # export directory for assets
             --pred-dir /tmp/ymir/export/export-0/annotations \ # export directory for annotations
             --media-location ~/ymir-assets \ # assets storage directory
             --src-revs mining-0@mining \
             --asset-format raw \ # export raw images
             --anno-format none # no annotations needed
$ find /tmp/ymir/export/export-0/assets > /tmp/ymir/export/export-0/index.tsv

After the export is done, users can see images at /tmp/ymir/export/export-0/assets. Users can send these pictures for annotation, and the annotations need to be saved in VOC format (assuming the save path is still /tmp/ymir/export/export-0/annotations). Once the annotation is finished, the user can import the data using a similar approach to the import command in 4.2.2.

$ mir import --index-file /tmp/ymir/export/export-0/index.tsv
             --gt-dir /tmp/ymir/export/export-0/annotations \ # annotation directory
             --gen-dir ~/ymir-assets \ # asset storage directory
             --unknown-types-strategy stop \
             --anno-type det-box \
             --dst-rev 'labeled-0@import' # destination branch and task name

The branch "labeled-0" now contains the 200 new training images. Users can be merged together with the original training set by the merge command.

$ mir merge --src-revs tr-va@filtered;tr:labeled-0@import \ # source branch
            --dst-rev tr-va-1@merged \ # destination branch and task name
            -s host

Now branch tr-va-1 contains the previous training and validation set and 200 new images we have just mined and labeled. A new model can be trained on this set with the following command:

$ mir train -w /tmp/ymir/training/train-1 \ # use different working dir for training and mining task
            --media-location ~/ymir-assets \
            --model-location ~/ymir-models \
            --task-config-file ~/training-config.yaml \
            --src-revs tr-va-1@merged \ # use new-merged branch
            --dst-rev training-1@trained \
            --executor youdaoyzbx/ymir-executor:ymir1.3.0-yolov5-cu111-tmi

YMIR-command-api.211028

Common parameter format and definitions

mir init

mir branch

The YMIR project consists of three components:

1. Back-end, task distribution and management.

2. Front-end, interactive interface.

3. Command line, a command-line interface (CLI) for managing the underlying annotation and image data.

Any code in the YMIR repo should follow the coding standards and will be checked in the CI tests.

• Functional code needs to be unit tested.

• Use flake8 or black to format the code before committing. Both of these follow the PEP8 and Google Python Style style guides.

• mypy - Python must go through static type checking.

Also check out MSFT Encoding Style for more advice.

Check this document for details

The dataset consists of metadata and media files, and the metadata has the following characteristics:

• A unique ID and the system has an initial default metadata status of null.

• A list of resources, where each element points to an actual resource; Metadata doesn't actually hold resources, but only maintains this list of resources.

• A number of keywords by which a user can search for a particular metadata status.

• Support users to create a new metadata branch and perform operations on the newly created branch. The operations on the new branch do not affect the status of the original metadata, and the original metadata is still traceable by the user. These operations include but are not limited to the following:

  (1) Adding resources (2) Adding or modifying annotations (3) Add or modify keywords (4) Filtering resources (5) Merging two different metadatas

• You can switch freely between different metadata.

• You can query the history of the metadata.

• You can tag the metadata to facilitate precise search by tag.

• You can also add keywords to metadata to facilitate fuzzy search through keywords.

• You can read the resources contained in a metadata and use those resources for browsing, training and so on.

From the above description, it can be seen that the management of metadata is similar to that of VCS (Version Control System), and users can have the following completely different usage methods and scenarios:

The first scene: Directly from the very first metadata, a filtering process is carried out to select and use the data that meets the requirements, as shown in the following figure:

Whenever the user needs to start a new task, :: The user checks out a new feature branch from within the current master branch, getting the metadata in feature#1 state. :: The user performs data filtering and other tasks on the metadata of this new branch. The user can obtain the metadata in the feature #2 state. :: When it is confirmed that this metadata is suitable for the user's training task, then the user can start training using this data.

• At this point, changes made by other users to the master branch's metadata will not affect the training data the user is using either.

The second scene: Search for certain metadata by label or keyword. The user starts the screening process until the data meets the requirements, and then the data is used. As shown below:

At this point, whenever a user needs to carry out a new task, :: Users can search for metadata that basically matches the user's requirements by means of keywords, labels, and so on. :: On this basis, users need sign out a new branch. :: Users can continue data filtering or cleansing on the new branch to obtain data that actually meets the requirements. :: Users can use this data for training.

The third scene: incremental merging. Suppose the user has completed the training task of the model using certain metadata. At this point, there is an update to the metadata of the repository and the master branch. The user wishes to merge this part of the update into the currently used metadata.

Suppose the user is now in FEATURE#2 and needs to do the following: :: You need switch back to master branch master. :: You need repeat the task previously done for the incremental part master#2 - master#1 to obtain feature#2+. :: You need cut back to feature#2 and merge feature#2+ to get feature#3.

The discussion in this section is based on the following assumptions: :: The user's data is imported in batches in units of datasets. :: Each dataset is a separate branch. :: Changes to and maintenance of each dataset are carried out on this branch. :: Master branch is always empty. This management approach is shown in the following figure:

We use the concept of code version control in Git to manage our data and models, use the concept of branches to create new projects so that different tasks on the same set of images can run in parallel. The additions, retrievals, updates, and deletions of datasets and basic operations are created by commits to branches. Logically, each commit stores an updated version of the dataset or new model, as well as the metadata of the operation that led to this change. Finally, only the data changes are merged into the main branch. This branch conceptually aggregates all the data annotated by many projects on the platform.

Why did the upload of the local dataset fail?

Regardless of whether the dataset has a label file, the images folder and annotations folder must be created. The images are placed in the images folder and the format is limited to jpg, jpeg, and png. The annotation files are placed in the annotations folder and the format is the pascal (when there is no annotation file, the folder is empty). Please put the images and annotations in the same folder and compress them into a ".zip" compressed package (not a .rar compressed format).

How should I obtain training and mining configuration files?

The default profile template needs to be extracted in the mirror.

The training image industryessentials/executor-det-yolov4-training:release-0.1.2 has a configuration file template located at: /img-man/training-template.yaml

Mining and inference mirrors The configuration file templates for industryessentials/executor-det-yolov4-mining:release-0.1.2 are located at: /img-man/mining-template.yaml (mining) and /img-man/infer-template. yaml (infer)

How can the trained model be used outside the system?

After successful training, the system will output the ID of the model. The user can find the corresponding file according to this id at --model-location. In fact, it is a tar file that can be extracted directly using the tar command to get the "mxnet" model file in parameters and JSON format.

How to solve the deployment, debugging and operation problems encountered in windows system?

It has not been fully tested on Windows server, so we cannot provide service support for the time being.

How to import models I've already trained?

See this document