fcp-indi / cpac Goto Github PK

Describe the bug

1. Tests succeed with pytest --doctest-modules --platform singularity locally
2. Tests fail with pytest --doctest-modules --platform singularity on GitHub Actions
   1. without --cleanenv:

      FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: unable to create new build: failed to find mount point for $GITHUB_WORKSPACE/.singularity/tmp: no parent mount point found

   2. with --cleanenv:

      FATAL:   Couldn't not parse configuration file /opt/hostedtoolcache/singularity/3.7.1/x64/etc/singularity/singularity.conf: open /opt/hostedtoolcache/singularity/3.7.1/x64/etc/singularity/singularity.conf: no such file or directory

To reproduce

In a branch or fork, add singluarity back to

Expected behavior

Singularity image builds and runs on GitHub Actions

Acceptance criteria

• Singularity tests that pass locally also pass on GitHub Actions

Screenshots

No response

C-PAC version

v1.8.3

Container platform

Singularity

Docker and/or Singularity version(s)

3.6.4, 3.7, 3.8

Additional context

No response

Describe the bug

Giving cpac options (e.g., --platform and --image) after the command (e.g., run, utils), causes those options to be ignored or used incorrectly.

To Reproduce
Steps to reproduce the behavior:
Take a set of options that works, e.g.,

cpac --platform singularity utils --help

and move the cpac option(s) to the end

cpac utils --help --platform singularity

Expected behavior

The options work the same regardless of placement or message clarifies the expected sequence or grouping of options.

Versions

• cpac: 0.3.0
• Container Platform :
  • Docker
  • Singularity

Related problem

Apptainer v1.0 is the Linux-official release after Singularity 3.8, but Sylabs is keeping a divergent Singularity growing with 3.9

Proposed feature

Keep the Singularity backend but add an Apptainer backend with the same/comparable functionality

Acceptance criteria

• supports Singularity 3.8
• supports Apptainer 1.0
• supports Singularity 3.9

Alternatives

Pick between Apptainer and Sylabs Singularity and only support one

Additional context

No response

Related problem

Right now, if --platform singularity is given without an image, cpac just pulls without even trying to find a local image.

Proposed feature

A configuration file that specifies the path to a default Singularity image seems like the most obvious solution.

Alternatives

• A default path?
• An environment variable?

Describe the bug

docker.errors.APIError: 400 Client Error for http+docker://localhost/v1.40/containers/create: Bad Request ("Duplicate mount point: /home/jclucas/CNL/GitHub/C-PAC")

To Reproduce
Steps to reproduce the behavior:

1. run cpac -B $PWD:/code $BIDS_DIR $OUTPUTS test_config
2. See error

Expected behavior

cpac would bind the custom binding in place of the automatic binding (i.e.,

Loading 🐳 Docker
Loading 🐳 fcpindi/c-pac:latest with these directory bindings:
  local                                Docker                       mode
  -----------------------------------  ---------------------------  ------
  /home/jclucas/CNL/GitHub/C-PAC       /code                        rw

instead of trying to bind both

Loading 🐳 Docker
Loading 🐳 fcpindi/c-pac:latest with these directory bindings:
  local                                Docker                           mode
  -----------------------------------  ------------------------------  ------
  /home/jclucas/CNL/GitHub/C-PAC       /home/jclucas/CNL/GitHub/C-PAC  rw
  /home/jclucas/CNL/GitHub/C-PAC       /code                           rw

)

Screenshots

Versions

• C-PAC: 1.8.0
• Container Platform :
  • Docker: 20.10.2

Additional context

Describe the bug

Direct URL dependencies

⚠️ Attention

PyPI and other standards-conformant package indices do not accept packages that declare dependencies using direct URLs. pip will accept them when installing packages from the local filesystem or from another URL, however.

― Dependencies Management in Setuptools: Direct URL dependencies

Until ba-timeseries-gradients and tsconcat publish to PyPI, we can't publish with them as dependencies

Expected behavior

cpac updates can be published to PyPI

Acceptance criteria

Either

• both of these new dependencies get published to PyPI or are containerized,
  • ba-timeseries-gradients
    • childmindresearch/ba-timeseries-gradients#14
    • ghcr.io/cmi-dair/ba-timeseries-gradients:main
  • ba-tsconcat
    • ba-tsconcat published on PyPI
• we install them in the C-PAC container and wrap them via that container with this wrapper,
• we reconsider #39,
• we pull cpac from PyPI and require users to install from GitHub,
  or
• we require users to install those dependencies from GitHub if they want to use them

Screenshots

with direct URL dependencies

without direct URL dependencies

Additional context

• PyPI Doesn't Allow Git Repo Dependencies

Project sub-parts and estimates

• Call ba_timeseries_gradients from cpac ① (#42)
• Call tsconcat from cpac ① (#43)

Timeline notes

version management

• ba_timeseries_gradients requires Python ≥ 3.11, < 3.12
• tsconcat requires Python ≥ 3.11

Frustration notes

I think the risk : reward and cost : benefit ratios are too high for these features, even though the risks and costs are low. Adding other scripts to be used directly as-is at the same level of abstraction without any other integration adds opportunity for error (more places for typos, more code and documentation to maintain) without any added value that I can see.

Design decisions and notes

I think I'll add a --version flag to each of the newly wrapped commands to specify a version of the wrapped packages, defaulting to the most recent release.

Describe the bug

Traceback (most recent call last):
  File "/home/jclucas/miniconda3/bin/cpac", line 8, in <module>
    sys.exit(run())
  File "/home/jclucas/miniconda3/lib/python3.7/site-packages/cpac/__main__.py", line 397, in run
    main(parsed)
  File "/home/jclucas/miniconda3/lib/python3.7/site-packages/cpac/__main__.py", line 307, in main
    Backends(**arg_vars).run(
  File "/home/jclucas/miniconda3/lib/python3.7/site-packages/cpac/backends/__init__.py", line 27, in Backends
    }[platform](**kwargs)
  File "/home/jclucas/miniconda3/lib/python3.7/site-packages/cpac/backends/singularity.py", line 30, in __init__
    kwargs = self.collect_config_bindings(self.config, **kwargs)
  File "/home/jclucas/miniconda3/lib/python3.7/site-packages/cpac/backends/platform.py", line 116, in collect_config_bindings
    inner_binding = self._collect_config_binding(config, c_b[0])
  File "/home/jclucas/miniconda3/lib/python3.7/site-packages/cpac/backends/platform.py", line 96, in _collect_config_binding
    return config.get('pipeline_setup', {}).get(config_key, {}).get('path')
AttributeError: 'list' object has no attribute 'get'

To reproduce

1. Create or acquire a minimal C-PAC pipeline config (one that begins with FROM: )
2. Run cpac $WRAPPER_OPTIONS run $BIDS_DIR $OUTPUT_DIR participant --pipeline_file $MINIMAL_CONFIG
3. See error

Expected behavior

cpac would pass the minimal config to C-PAC which would run normally.

Acceptance criteria

• cpac does not crash when given a minimal pipeline config

C-PAC version

v1.8.1-dev

Container platform

Docker, Singularity

Describe the bug

UnicodeEncodeError: 'latin-1' codec can't encode character '\u24c8' in position 8: ordinal not in range(256)

Versions

• cpac: 0.3.0
• Container Platform :
  • Singularity

Hi all,

I am currently trying to try to run the default C-PAC pipeline through a Docker container using C-PAC Python package. However, after installing C-PAC and Docker, I am running into the error - "ModuleNotFoundError: No module named 'pwd' " with stack trace:

(venv) C:\Users\Evan\PycharmProjects\DataScienceProject\venv\TDAfMRI>cpac run --help
Traceback (most recent call last):
File "C:\Users\Evan\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 194, in run_module_as_main
return run_code(code, main_globals, None,
File "C:\Users\Evan\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Users\Evan\PycharmProjects\DataScienceProject\venv\Scripts\cpac.exe_main.py", line 7, in
File "c:\users\evan\pycharmprojects\datascienceproject\venv\lib\site-packages\cpac_main.py", line 397, in run
main(parsed)
File "c:\users\evan\pycharmprojects\datascienceproject\venv\lib\site-packages\cpac_main.py", line 307, in main
Backends(**arg_vars).run(
File "c:\users\evan\pycharmprojects\datascienceproject\venv\lib\site-packages\cpac\backends_init_.py", line 20, in Backends
from .docker import Docker
File "c:\users\evan\pycharmprojects\datascienceproject\venv\lib\site-packages\cpac\backends\docker.py", line 6, in
from cpac.backends.platform import Backend, Platform_Meta
File "c:\users\evan\pycharmprojects\datascienceproject\venv\lib\site-packages\cpac\backends\platform.py", line 3, in
import pwd
ModuleNotFoundError: No module named 'pwd'

Does anyone understand this error and how it can be avoided?

Version Details

• Windows 10 x64 system
• Python 3.8.8
• Docker version 20.10.5, build 55c4c88

Related problem

It's hard to talk about this tool (cpac) and the tool it wraps (C-PAC) without being super wordy.

Proposed feature

We rename this tool cpac-wrapper or cpac-cli or somesuch

Acceptance criteria

• https://github.com/FCP-INDI/cpac and https://github.com/FCP-INDI/C-PAC no longer have the same initialism as their names

Alternatives

We move this project into the main repo and package them together?

Additional context

• When we did an initial evaluation of C-PAC for NMIND, people seemed very confused about how cpac and C-PAC would have different versions (0.4.0 and 1.8.3, respectively).
• The is something of a revisit of shnizzedy/cpac#2

Related problem

When I linted the codebase in #40, I ignored the missing docstrings

Proposed feature

Add all the missing docstrings, at least on public modules / functions / classes / methods

Acceptance criteria

• no missing docstrings!

Alternatives

No response

Additional context

No response

Related problem

Currently to upgrade a Docker or Singularity image, the user must do so outside of cpac.

Proposed feature

Add a pull or upgrade command to pull the latest container with the default or given tag.

Describe the bug

cpac run --help runs into OSError: [Errno 30] Read-only file system: '/logs' after attempting to load docker. It is not always /logs but can be other folders such as /crash, it appears to be random.
Full output from running cpac run --help:

Loading 🐳 Docker
Traceback (most recent call last):
File "/opt/miniconda3/bin/cpac", line 8, in
sys.exit(run())
File "/opt/miniconda3/lib/python3.9/site-packages/cpac/main.py", line 397, in run
main(parsed)
File "/opt/miniconda3/lib/python3.9/site-packages/cpac/main.py", line 307, in main
Backends(**arg_vars).run(
File "/opt/miniconda3/lib/python3.9/site-packages/cpac/backends/init.py", line 24, in Backends
{
File "/opt/miniconda3/lib/python3.9/site-packages/cpac/backends/docker.py", line 33, in init
self._collect_config(**kwargs)
File "/opt/miniconda3/lib/python3.9/site-packages/cpac/backends/docker.py", line 68, in _collect_config
kwargs = self.collect_config_bindings(self.config, **kwargs)
File "/opt/miniconda3/lib/python3.9/site-packages/cpac/backends/platform.py", line 125, in collect_config_bindings
os.makedirs(inner_binding, exist_ok=True)
File "/opt/miniconda3/lib/python3.9/os.py", line 225, in makedirs
mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/logs'

To reproduce

1. MacOS Monterey 12.0.1
2. Install Docker for Mac as described on Docker website
3. pip install cpac (is successful)
4. cpac run --help --> throws error as shown

Expected behavior

cpac run --help runs as expected without this error.

Acceptance criteria

cpac run --help runs without error. Note: cpac --help runs just fine. Other docker image wrappers and pulls work fine too (e.g. docker pull fmriprep works, and fmriprep-docker wrapper --help works without error as well).

Additionally cpac --version does not return version, but rather returns the whole help documentation for cpac. Same as cpac --help, although cpac run --version does give version number.

Screenshots

No response

C-PAC version

0.3.2.post1

Container platform

Docker

Docker and/or Singularity version(s)

Docker 20.10.10, build b485636

Additional context

As mentioned briefly before, interestingly cpac --help works, but cpac --version fails and just gives the cpac --help output. Meanwhile cpac run --help gives the error as reported in this post, but cpac run --version gives a version number just fine.

Related problem

Right now, cpac read-only binds the directory containing a user-provided data config file

이지원 /cpax_forum: Re: RuntimeError

Proposed feature

Just bind the data config file itself, something like

 dc_file = os.path.abspath(kwargs['data_config_file']) 
 self._bind_volume(dc_file, dc_file, 'r') 
 locals_from_data_config = Locals_to_bind() 
 locals_from_data_config.from_config_file( 
     dc_file 
 ) 

Acceptance criteria

• data config file binds read-only but doesn't (necessarily) bind its parent directory read-only

Alternatives

No response

Additional context

Maybe also bind individual files from the data config instead of their parent directories too?

Describe the bug

Additionally cpac --version does not return version, but rather returns the whole help documentation for cpac. Same as cpac --help, although cpac run --version does give version number.

― #21 (comment)

To reproduce

1. Run cpac --version
2. See usage information

Expected behavior

cpac --version returns version information

Acceptance criteria

• cpac --version returns the version of cpac

Screenshots

No response

C-PAC version

No response

Container platform

No response

Docker and/or Singularity version(s)

No response

Additional context

cpac version 0.4.0

Related problem

cpac creates directories that it expects before binding them (so the directories will be created/owned by the user running cpac), but sometimes those directories are empty by the time cpac is done, especially if running utilities.

Proposed feature

Keep track of directories created by cpac, and in the cleanup phase, delete any that are empty.

Acceptance criteria

• empty directories created by cpac do not persist once cpac exits

Alternatives

No response

Additional context

No response

Because the image is huge, takes too long to pull. I would suggest using fake images, with the same interface as C-PAC.

Originally posted by @anibalsolon in #1 (comment)

Describe the bug

The option is listed in the README as

  -o [OPT [OPT ...]], --container_option [OPT [OPT ...]]
                        parameters and flags to pass through to Docker or Singularity

                        This flag can take multiple arguments so cannot be
                        the final argument before the command argument (i.e.,
                        run or any other command that does not start with - or --)

defined here

but I don't see that parameter handled anywhere

Passing options through don't seem to work with any syntax I've attempted (see "To reproduce" below).

To reproduce

cpac -o --entrypoint=/bin/bash --platform docker run

cpac: error: argument -o/--container_option: expected one argument

cpac -o "--entrypoint=/bin/bash" --platform docker run

cpac: error: argument -o/--container_option: expected one argument

cpac -o="--entrypoint=/bin/bash" --platform docker run

cpac run: error: the following arguments are required: bids_dir, output_dir, level_of_analysis, extra_args

cpac -o="--entrypoint=/bin/bash" --platform docker run bids_dir outputs_dir participant

run.py: error: unrecognized arguments: -o=--entrypoint=/bin/bash

Expected behavior

Any options passed to -o or --container_option are passed through to Docker or Singularity.

Acceptance criteria

• cpac -o="--entrypoint=/bin/bash" --platform docker run opens a BASH prompt in a C-PAC Docker container

cpac version

0.3.2.post1

C-PAC version

v1.8.1, v1.8.1-dev

Container platform

Docker, Singularity

Describe the bug

Could not find the file /cpac_resources/default_pipeline.yml in container

To reproduce

Hi All,
I tried to run cpac using the following command

cpac --platform docker --tag nightly run /home/user3/mox_study/bids_mox /home/user3/mox_study/cpacoutput participant --participant_label mox2769 --working_dir /home/user3/mox_study/cpacwork --preconfig preproc --vv

However, it gave me the error below. Ive tried various other preconfig pipelines and cpac versions but i still get the same error. I installed the latest cpac wrapper via pip 3.6 earlier.

Error:

Loading 🐳 Docker
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/c1b5a72741e24b5eabee508ca854a27d2d5b7db3a2d9a3b9ee9647ba58fb30e5/archive?path=%2Fcpac_resources%2Fdefault_pipeline.yml

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/cpac", line 8, in <module>
    sys.exit(run())
  File "/usr/local/lib/python3.6/site-packages/cpac/__main__.py", line 403, in run
    main(parsed)
  File "/usr/local/lib/python3.6/site-packages/cpac/__main__.py", line 307, in main
    Backends(**arg_vars).run(
  File "/usr/local/lib/python3.6/site-packages/cpac/backends/__init__.py", line 27, in Backends
    }[platform](**kwargs)
  File "/usr/local/lib/python3.6/site-packages/cpac/backends/docker.py", line 33, in __init__
    self._collect_config(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/cpac/backends/docker.py", line 61, in _collect_config
    stream = container.get_archive(path=self.pipeline_config)[0]
  File "/usr/local/lib/python3.6/site-packages/docker/models/containers.py", line 262, in get_archive
    chunk_size, encode_stream)
  File "/usr/local/lib/python3.6/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/docker/api/container.py", line 748, in get_archive
    self._raise_for_status(res)
  File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.41/containers/c1b5a72741e24b5eabee508ca854a27d2d5b7db3a2d9a3b9ee9647ba58fb30e5/archive?path=%2Fcpac_resources%2Fdefault_pipeline.yml: Not Found ("Could not find the file /cpac_resources/default_pipeline.yml in container c1b5a72741e24b5eabee508ca854a27d2d5b7db3a2d9a3b9ee9647ba58fb30e5")

Preconfig

• default
• abcd-options
• anat-only
• blank
• ccs-options
• fmriprep-options
• fx-options
• monkey
• monkey-ABCD
• ndmg
• nhp-macaque
• preproc
• rbc-options
• rodent

Expected behavior

..

Acceptance criteria

..

C-PAC version

nightly

Container platform

Docker

Docker and/or Singularity version(s)

Docker version 20.10.18, build b40c2f6

Describe the bug

 Traceback (most recent call last):
  File "/share/█████████/apps/conda/bin/cpac", line 8, in <module>
    sys.exit(run())
  File "/share/█████████/apps/conda/lib/python3.7/site-packages/cpac/__main__.py", line 310, in run
    main(sys.argv)
  File "/share/█████████/apps/conda/lib/python3.7/site-packages/cpac/__main__.py", line 290, in main
    Backends(**arg_vars).run(
  File "/share/█████████/apps/conda/lib/python3.7/site-packages/cpac/backends/__init__.py", line 27, in Backends
    }[platform](**kwargs)
KeyError: None 

To Reproduce
Run cpac run with --platform option after run.

Expected behavior

--platform option is respected or a message tells the user where to use the --platform option.

Versions

• C-PAC: 0.3.0
• Container Platform :
  • Docker
  • Singularity

Additional context
Related: #2

Describe the bug

4. In this figure, is the /media/12T/practice is read-write and the /media, /media/12T directories are read-only? So the permission was denied?
   
   ― 이지원 /cpax_forum: Re: RuntimeError

That's a bug! It looks like the utils data_config build command writes to outputSubjectListLocation from DATA_SETTINGS_FILE (

Usage: cpac utils data_config build [OPTIONS] DATA_SETTINGS_FILE

), but cpac isn't automatically binding that directory. In this example, /media/12T in your container doesn't point to anywhere, so you can't write to that path (/media/12T/practice in your container points to /media/12T/practice locally, but the parent directories aren't accessible from the container). As a stopgap until the issue is resolved, you could pass the outputSubjectListLocation file or directory as a custom binding or as the working directory, or run from the outputSubjectListLocation directory.
― @shnizzedy /cpax_forum: Re: RuntimeError

To reproduce

No response

Expected behavior

cpac binds the output directory automatically

Acceptance criteria

• cpac binds the output directory automatically

Screenshots

No response

C-PAC version

C-PAC v1.8.3, cpac v0.4.0

Container platform

No response

Docker and/or Singularity version(s)

No response

Additional context

No response

Related problem

I want to use the C-PAC command-line interface (CLI) with ABIDE dataset.

data_setting.yml file is ready to run.

$ cpac utils data_config build ~/abide_data_setting.yml

After this command line I got an error as shown below.

How can I get data_config file without this error?

― 이지원: Error on data_setting.yml

Proposed feature

Automatically bind the path given to cpac utils data_config build in the data settings file under the key outputSubjectListLocation when loading the C-PAC container for utils data_config build

Acceptance criteria

• cpac automatically binds outputSubjectListLocation

Alternatives

No response

Additional context

No response