Squeezing peak performance out of the cloud CI about xmip HOT 7 CLOSED

Oh damn. I was wrong and @andersy005 was right. The xdist was not even activated. Squeezed a good 7 min of the runtime r🚀. See #86

from xmip.

Hooray🚀 Glad to hear that activating pytest-xdist shaves minutes off the CI runtime...

from xmip.

@jbusecke,

• I am seeing pytest-xdist listed in your conda environments, but I don't see it being used to run tests. Have you looked into it yet (e.g.: pytest -n auto --maxfail 5 --reruns 1 ....?

• I'd recommend using conda-forge channel only and setting the channel priority to strict. Also, the current conda setup configures mamba but it doesn't use it to setup the environment: https://github.com/jbusecke/cmip6_preprocessing/blob/a5ae25867afe509095de8757b5c1ba92192e0549/.github/workflows/ci.yaml#L34-L39

I'd try these settings to see if there is any improvements:

- uses: conda-incubator/setup-miniconda@v2
  with:
    channels: conda-forge
    mamba-version: '*'
    channel-priority: strict
    activate-environment: test_env_cmip6_preprocessing # Defined in ci/environment*.yml
    auto-update-conda: false
    python-version: ${{ matrix.python-version }}
    use-only-tar-bz2: true # IMPORTANT: This needs to be set for caching to work properly!
- name: Set up conda environment
  shell: bash -l {0}
  run: |
    mamba env update -f ci/environment-py${{ matrix.python-version }}.yml
    python -m pip install -e . --no-deps --force-reinstall
    conda list

from xmip.

Ah thanks a lot for the tips @andersy005 . I specify the -n auto in pyproject.toml I think. But the mamba setup is very new to me, so that makes a lot of sense.

from xmip.

It seems like the env update ... call is already using mamba in my configuration here. That seems to be equivalent to the mamba env update -f ci/environment-py${{ matrix.python-version }}.yml you proposed?

from xmip.

Or is there something I am missing.

Unfortunately setting the channel priority did not decrease the build time significantly either. I am however getting this weird message, that indicates that something is not set up quite right with the channels. I am not sure to which side (the ci.yaml configuration or the env file) the workflow resolves...

Wondering if using minimamba instead of miniconda would speed this up (since that one should not have to solve for a bunch of packages to install mamba?)

from xmip.

Ill close this for now.

from xmip.