iterative / cml_dvc_case Goto Github PK

As a remote dir I use GDrive.

When I pull the data to my personal computer it requires manual authentication conducted by clicking the provided link.

When I try to use a similar action as you do I get error cause remote computer on which github tries to run action can't conduct an authentification in this way.

Is there a possibility to authenticate gdrive in a smillar way as you do withn S3 bucket - with some kind of token?

Hi,

First of all, thank you for the nice tools developed by you. I am trying to create a training ML workflow with gitlab, CML and DVC with MinIO storage as my remote backup where I have my training dataset stored. my .gitlab-ci.yaml looks like this:

stages:
  - cml_run
cml:
  stage: cml_run
  image: dvcorg/cml:0-dvc2-base1-gpu

  script:
  - echo 'Hi from CML' >> report.md
  - apt-get update && apt-get install -y python3-opencv
  - pip3 install -r requirements.txt
  - dvc remote add -d minio_data s3://bucket/dataset/
  - dvc remote modify minio_data endpointurl http://<MINIOSERVER_IP_ADDRESS>:9000
  - dvc remote modify minio_data use_ssl False
  - export AWS_ACCESS_KEY_ID="xxxxxxx"
  - export AWS_SECRET_ACCESS_KEY="xxxxxxx"
  - dvc pull -r minio_data
  - python main.py
  - cml-send-comment report.md --repo=https://<my_gitlab_repo_url>

My setup is configured as following:

• A gitlab self-hosted runner listening for job (works: Ubuntu 20.04, 2 x RTX 3070 GPUs, ).
• An S3 MinIO storage server configured as DVC remote local backup (works with my credentials).
• A training script (works).

My workflow is working and I am able to train my model on the runner and queue jobs but I have the following issues with it (maybe there is a better way to do this, hence I am here asking for directions):

1. For each training job, the entire dataset is pulled from the remote and then the model is trained. This is really slow. It is my requirement to keep using dvc for data versioning but is there a way to bypass the dataset pull dvc pull -r minio_data everytime and use the same data between different training jobs? (maybe mount volumes to the docker container?)
2. For MinIO authentication, I do not want to put my credentials as in AWS_SECRET_ACCESS_KEY in the .gitlab-ci.yaml file, in case more than one person want to use this workflow to queue their training jobs in a collaborative environment. What other options do I have?
3. Is there a way to configure a local container registry cache for the runner (and this worflow) where I can put all the necessary docker images and use them instead of adding dependencies to the workflow like I am doing and let docker handle it?

Any feedback or suggestions would be appreciated. Thank you.

Hello,

I am trying to reproduce the test case but I am unable to include plot images in my report it throws this error

"""
{"code":"ERR_INVALID_URL","input":"\r\n<title>400 Bad Request</title>\r\n\r\n

Without images the report looks good, here is my cml.yaml file

name: train-my-model
on: [push]

jobs:
run:
runs-on: [ubuntu-latest]

  steps:
    - uses: actions/checkout@v3
      with:
        ref: ${{ github.event.pull_request.head.sha }}

    - uses: iterative/setup-cml@v1

    - uses: iterative/setup-dvc@v1

    - uses: actions/setup-python@v2
      with:
        python-version: '3.x'

    - name: cml
      env:
        repo_token: ${{ secrets.REPO_SECRET }}
      run: |
        pip install -r requirements.txt

        # Pull dataset with DVC
        #dvc pull data

        # Reproduce pipeline if any changes detected in dependencies
        #dvc repro

        # Use DVC metrics diff to compare metrics to master
        git fetch --prune --unshallow
        dvc metrics diff master >> report.md

        # Add figure to report
        dvc plots diff --target loss.csv --show-vega master > vega.json
        vl2png vega.json -s 1.3 > vega.png
        echo '![](./vega.png)' >> report.md
        cml comment create --pr --publish report.md