Comments (4)
matthewfeickert
commented on June 16, 2024
While you would normally want to use the 'local' extra
python -m pip install --upgrade 'recast-atlas[local]'
for the local backend option, I can reproduce your bug even with it. I think this is another case of yadage/yadage#116 that is getting hit, as
recast run examples/rome --backend docker
runs without errors.
Run output:
$ recast run examples/rome --backend docker
2023-11-04 04:15:18,036 | packtivity.asyncback | INFO | configured pool size to 16
2023-11-04 04:15:18,371 | yadage.creators | INFO | initializing workflow with initdata: {'did': 404958, 'dxaod_file': 'https://recastwww.web.cern.ch/recastwww/data/reana-recast-demo/mc15_13TeV.123456.cap_recast_demo_signal_one.root', 'xsec_in_pb': 0.00122} discover: True relative: True
2023-11-04 04:15:18,371 | adage.pollingexec | INFO | preparing adage coroutine.
2023-11-04 04:15:18,371 | adage | INFO | starting state loop.
2023-11-04 04:15:18,458 | yadage.wflowview | INFO | added </init:0|defined|unknown>
2023-11-04 04:15:19,438 | yadage.wflowview | INFO | added </eventselection:0|defined|unknown>
2023-11-04 04:15:20,710 | yadage.wflowview | INFO | added </statanalysis:0|defined|unknown>
2023-11-04 04:15:22,162 | adage.pollingexec | INFO | submitting nodes [</init:0|defined|known>]
2023-11-04 04:15:22,666 | pack.init.step | INFO | publishing data: <TypedLeafs: {'did': 404958, 'dxaod_file': 'https://recastwww.web.cern.ch/recastwww/data/reana-recast-demo/mc15_13TeV.123456.cap_recast_demo_signal_one.root', 'xsec_in_pb': 0.00122}>
2023-11-04 04:15:22,666 | adage | INFO | unsubmittable: 0 | submitted: 0 | successful: 0 | failed: 0 | total: 3 | open rules: 0 | applied rules: 3
2023-11-04 04:15:24,248 | adage.node | INFO | node ready </init:0|success|known>
2023-11-04 04:15:24,248 | adage.pollingexec | INFO | submitting nodes [</eventselection:0|defined|known>]
2023-11-04 04:15:24,250 | pack.eventselection. | INFO | starting file logging for topic: step
2023-11-04 04:15:36,945 | adage.node | INFO | node ready </eventselection:0|success|known>
2023-11-04 04:15:36,945 | adage.pollingexec | INFO | submitting nodes [</statanalysis:0|defined|known>]
2023-11-04 04:15:36,947 | pack.statanalysis.st | INFO | starting file logging for topic: step
2023-11-04 04:15:42,039 | adage.node | INFO | node ready </statanalysis:0|success|known>
2023-11-04 04:15:42,059 | adage.controllerutil | INFO | no nodes can be run anymore and no rules are applicable
2023-11-04 04:15:42,059 | adage.controllerutil | INFO | no nodes can be run anymore and no rules are applicable
2023-11-04 04:15:42,060 | adage | INFO | unsubmittable: 0 | submitted: 0 | successful: 3 | failed: 0 | total: 3 | open rules: 0 | applied rules: 3
2023-11-04 04:15:45,620 | adage | INFO | adage state loop done.
2023-11-04 04:15:45,620 | adage | INFO | execution valid. (in terms of execution order)
2023-11-04 04:15:45,620 | adage | INFO | workflow completed successfully.
2023-11-04 04:15:45,620 | yadage.steering_api | INFO | done. dumping workflow to disk.
2023-11-04 04:15:45,622 | yadage.steering_api | INFO | visualizing workflow.
2023-11-03 23:15:46,306 | recastatlas.subcomma | INFO | RECAST run finished.
RECAST result examples/rome recast-c2b6ce7a:
--------------
- name: CLs 95% based upper limit on poi
value:
exp: 0.8924846399964371
exp_m1: 0.6377501820447065
exp_m2: 0.4731739008380644
exp_p1: 1.2720762961732819
exp_p2: 1.7545752712294322
obs: 1.3352971254860764
- name: CLs 95% at nominal poi
value:
exp: 0.25999040745937085
exp_m1: 0.10547655600578199
exp_m2: 0.03889040527686523
exp_p1: 0.5345040672498215
exp_p2: 0.8276574946063575
obs: 0.574709475331039
which means that some dependencies have a bug that needs to get fixed. This isn't getting caught properly in our CI at the moment, so this will probably take some debugging early next week.
from recast-atlas.
matthewfeickert
commented on June 16, 2024
Ah, @Nollde no this is actually a simple permissions issue and lack of documentation at the end of it, in addition to yadage/yadage#116. I should have debugged better.
Okay, starting from scratch in a Python 3.11 environment as I can't claim that Python 3.12 will have support for all dependencies yet.
$ mkdir -p /tmp/debug && cd /tmp/debug
$ python -m venv recast-atlas-issue-118 && . recast-atlas-issue-118/bin/activate
(recast-atlas-issue-118) $ python -m pip install --upgrade pip setuptools wheel
(recast-atlas-issue-118) $ python -m pip install 'recast-atlas[local]' 'yadage<0.21.0' # c.f. https://github.com/yadage/yadage/issues/116If we then run
recast run examples/rome --backend local --tag debug
the jobs will start but then crash. That's strange as we already saw that running with the docker backend works fine. If we investiage the log files by checking out recast-debug/eventselection/_packtivity/eventselection.run.log, we'll see that we get a permissions error
...
2023-11-04 01:14:50,653 | pack.eventselection. | INFO | b'/build2/atnight/localbuilds/nightlies/21.2/AnalysisBase/athena/PhysicsAnalysis/D3PDTools/EventLoop/Root/Driver.cxx:107:exception: could not create output directory /tmp/debug/recast-debug/eventselection/submitDir'
...
that seems strange until we remember that recast-atlas currently assumes that you're root for all operations and so if you're not then you're going to need to tell it that you want it to run the containers as root. If you run
recast run examples/rome --backend docker --tag debug-helper
then we see that all the docker run commands have -u root added into them
$ grep "docker run" recast-debug-helper/eventselection/_packtivity/eventselection.step.log
docker run --rm -i --cidfile /tmp/debug/recast-debug-helper/eventselection/_packtivity/eventselection.cid -u root -v /tmp/debug/recast-debug-helper/eventselection:/tmp/debug/recast-debug-helper/eventselection:rw -v /tmp/debug/recast-debug-helper/init:/tmp/debug/recast-debug-helper/init:rw docker.io/reanahub/reana-demo-atlas-recast-eventselection:1.0 sh -c bash
2023-11-04 06:20:29,219 | pack.eventselection. | DEBUG | moving on from subprocess execution: docker run --rm -i --cidfile /tmp/debug/recast-debug-helper/eventselection/_packtivity/eventselection.cid -u root -v /tmp/debug/recast-debug-helper/eventselection:/tmp/debug/recast-debug-helper/eventselection:rw -v /tmp/debug/recast-debug-helper/init:/tmp/debug/recast-debug-helper/init:rw docker.io/reanahub/reana-demo-atlas-recast-eventselection:1.0 sh -c bash
2023-11-04 06:20:29,219 | pack.eventselection. | DEBUG | finally for docker run --rm -i --cidfile /tmp/debug/recast-debug-helper/eventselection/_packtivity/eventselection.cid -u root -v /tmp/debug/recast-debug-helper/eventselection:/tmp/debug/recast-debug-helper/eventselection:rw -v /tmp/debug/recast-debug-helper/init:/tmp/debug/recast-debug-helper/init:rw docker.io/reanahub/reana-demo-atlas-recast-eventselection:1.0 sh -c bashas a result of this packtivity environment setting getting set in the recast-atlas Dockerfile
recast-atlas/docker/Dockerfile
Line 49 in b660442
(this isn't documented well).
With that in mind we just need to do the same locally with our local backend run
PACKTIVITY_DOCKER_CMD_MOD="-u root" recast run examples/rome --backend local --tag working
then things run as expected.
Run output:
$ PACKTIVITY_DOCKER_CMD_MOD="-u root" recast run examples/rome --backend local --tag working
2023-11-04 01:26:35,769 | packtivity.asyncback | INFO | configured pool size to 16
2023-11-04 01:26:36,085 | yadage.creators | INFO | initializing workflow with initdata: {'did': 404958, 'dxaod_file': 'https://recastwww.web.cern.ch/recastwww/data/reana-recast-demo/mc15_13TeV.123456.cap_recast_demo_signal_one.root', 'xsec_in_pb': 0.00122} discover: True relative: True
2023-11-04 01:26:36,085 | adage.pollingexec | INFO | preparing adage coroutine.
2023-11-04 01:26:36,085 | adage | INFO | starting state loop.
2023-11-04 01:26:36,107 | yadage.wflowview | INFO | added </init:0|defined|unknown>
2023-11-04 01:26:36,177 | yadage.wflowview | INFO | added </eventselection:0|defined|unknown>
2023-11-04 01:26:36,252 | yadage.wflowview | INFO | added </statanalysis:0|defined|unknown>
2023-11-04 01:26:36,326 | adage.pollingexec | INFO | submitting nodes [</init:0|defined|known>]
2023-11-04 01:26:36,352 | pack.init.step | INFO | publishing data: <TypedLeafs: {'did': 404958, 'dxaod_file': 'https://recastwww.web.cern.ch/recastwww/data/reana-recast-demo/mc15_13TeV.123456.cap_recast_demo_signal_one.root', 'xsec_in_pb': 0.00122}>
2023-11-04 01:26:36,353 | adage | INFO | unsubmittable: 0 | submitted: 0 | successful: 0 | failed: 0 | total: 3 | open rules: 0 | applied rules: 3
2023-11-04 01:26:36,456 | adage.node | INFO | node ready </init:0|success|known>
2023-11-04 01:26:36,456 | adage.pollingexec | INFO | submitting nodes [</eventselection:0|defined|known>]
2023-11-04 01:26:36,460 | pack.eventselection. | INFO | starting file logging for topic: step
2023-11-04 01:26:47,761 | adage.node | INFO | node ready </eventselection:0|success|known>
2023-11-04 01:26:47,761 | adage.pollingexec | INFO | submitting nodes [</statanalysis:0|defined|known>]
2023-11-04 01:26:47,765 | pack.statanalysis.st | INFO | starting file logging for topic: step
2023-11-04 01:26:52,524 | adage.node | INFO | node ready </statanalysis:0|success|known>
2023-11-04 01:26:52,544 | adage.controllerutil | INFO | no nodes can be run anymore and no rules are applicable
2023-11-04 01:26:52,544 | adage.controllerutil | INFO | no nodes can be run anymore and no rules are applicable
2023-11-04 01:26:52,544 | adage | INFO | unsubmittable: 0 | submitted: 0 | successful: 3 | failed: 0 | total: 3 | open rules: 0 | applied rules: 3
2023-11-04 01:26:55,982 | adage | INFO | adage state loop done.
2023-11-04 01:26:55,982 | adage | INFO | execution valid. (in terms of execution order)
2023-11-04 01:26:55,982 | adage | INFO | workflow completed successfully.
2023-11-04 01:26:55,982 | yadage.steering_api | INFO | done. dumping workflow to disk.
2023-11-04 01:26:55,984 | yadage.steering_api | INFO | visualizing workflow.
2023-11-04 01:26:56,073 | recastatlas.subcomma | INFO | RECAST run finished.
RECAST result examples/rome recast-working:
--------------
- name: CLs 95% based upper limit on poi
value:
exp: 0.8924846399964371
exp_m1: 0.6377501820447065
exp_m2: 0.4731739008380644
exp_p1: 1.2720762961732819
exp_p2: 1.7545752712294322
obs: 1.3352971254860764
- name: CLs 95% at nominal poi
value:
exp: 0.25999040745937085
exp_m1: 0.10547655600578199
exp_m2: 0.03889040527686523
exp_p1: 0.5345040672498215
exp_p2: 0.8276574946063575
obs: 0.574709475331039
So:
- There is still a bug with yadage/yadage#116
- Running with the
localbackend requires the'local'extra and using it isn't documented well. packtivityenvironmental options aren't documented well at all either.
from recast-atlas.
Nollde
commented on June 16, 2024
Hi @matthewfeickert , thanks for mentioning yadage/yadage#116, I was not aware of this issue.
With yadage<0.21.0 I can indeed use the local workflow.
The PACKTIVITY_DOCKER_CMD_MOD is not needed in my case.
You can see this in a small example here:
Example on local workflow
Environment Setup
(base) ➜ issue_118: conda create --name recast_118 python==3.11
Collecting package metadata (current_repodata.json): done
Solving environment: unsuccessful attempt using repodata from current_repodata.json, retrying with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 23.7.4
latest version: 23.10.0
Please update conda by running
$ conda update -n base -c defaults conda
Or to minimize the number of packages updated during conda update use
conda install conda=23.10.0
## Package Plan ##
environment location: /Users/dnoll/anaconda3/envs/recast_118
added / updated specs:
- python==3.11
The following NEW packages will be INSTALLED:
bzip2 pkgs/main/osx-arm64::bzip2-1.0.8-h620ffc9_4
ca-certificates pkgs/main/osx-arm64::ca-certificates-2023.08.22-hca03da5_0
libffi pkgs/main/osx-arm64::libffi-3.4.4-hca03da5_0
ncurses pkgs/main/osx-arm64::ncurses-6.4-h313beb8_0
openssl pkgs/main/osx-arm64::openssl-1.1.1w-h1a28f6b_0
pip pkgs/main/osx-arm64::pip-23.3-py311hca03da5_0
python pkgs/main/osx-arm64::python-3.11.0-hc0d8a6c_3
readline pkgs/main/osx-arm64::readline-8.2-h1a28f6b_0
setuptools pkgs/main/osx-arm64::setuptools-68.0.0-py311hca03da5_0
sqlite pkgs/main/osx-arm64::sqlite-3.41.2-h80987f9_0
tk pkgs/main/osx-arm64::tk-8.6.12-hb8d0fd4_0
tzdata pkgs/main/noarch::tzdata-2023c-h04d1e81_0
wheel pkgs/main/osx-arm64::wheel-0.41.2-py311hca03da5_0
xz pkgs/main/osx-arm64::xz-5.4.2-h80987f9_0
zlib pkgs/main/osx-arm64::zlib-1.2.13-h5a0b063_0
Proceed ([y]/n)? y
Downloading and Extracting Packages
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate recast_118
#
# To deactivate an active environment, use
#
# $ conda deactivate
(base) ➜ issue_118: conda activate recast_118
(recast_118) ➜ issue_118: python -m pip install --upgrade pip setuptools wheel
Requirement already satisfied: pip in /Users/dnoll/anaconda3/envs/recast_118/lib/python3.11/site-packages (23.3)
Collecting pip
Downloading pip-23.3.1-py3-none-any.whl.metadata (3.5 kB)
Requirement already satisfied: setuptools in /Users/dnoll/anaconda3/envs/recast_118/lib/python3.11/site-packages (68.0.0)
Collecting setuptools
Using cached setuptools-68.2.2-py3-none-any.whl.metadata (6.3 kB)
Requirement already satisfied: wheel in /Users/dnoll/anaconda3/envs/recast_118/lib/python3.11/site-packages (0.41.2)
Collecting wheel
Using cached wheel-0.41.3-py3-none-any.whl.metadata (2.2 kB)
Downloading pip-23.3.1-py3-none-any.whl (2.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 10.2 MB/s eta 0:00:00
Using cached setuptools-68.2.2-py3-none-any.whl (807 kB)
Using cached wheel-0.41.3-py3-none-any.whl (65 kB)
Installing collected packages: wheel, setuptools, pip
Attempting uninstall: wheel
Found existing installation: wheel 0.41.2
Uninstalling wheel-0.41.2:
Successfully uninstalled wheel-0.41.2
Attempting uninstall: setuptools
Found existing installation: setuptools 68.0.0
Uninstalling setuptools-68.0.0:
Successfully uninstalled setuptools-68.0.0
Attempting uninstall: pip
Found existing installation: pip 23.3
Uninstalling pip-23.3:
Successfully uninstalled pip-23.3
Successfully installed pip-23.3.1 setuptools-68.2.2 wheel-0.41.3
(recast_118) ➜ issue_118: python -m pip install 'recast-atlas[local]' 'yadage<0.21.0'
Collecting yadage<0.21.0
Using cached yadage-0.20.1-py3-none-any.whl
Collecting recast-atlas[local]
Using cached recast_atlas-0.3.0-py3-none-any.whl.metadata (4.4 kB)
Collecting click>=7.0 (from recast-atlas[local])
Using cached click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Collecting jsonschema>=3.0.0 (from recast-atlas[local])
Using cached jsonschema-4.19.2-py3-none-any.whl.metadata (7.9 kB)
Collecting pyyaml>=5.1 (from recast-atlas[local])
Using cached PyYAML-6.0.1-cp311-cp311-macosx_11_0_arm64.whl.metadata (2.1 kB)
Collecting yadage-schemas>=0.10.7 (from recast-atlas[local])
Using cached yadage_schemas-0.11.0-py3-none-any.whl (20 kB)
Collecting pydotplus==2.0.2 (from recast-atlas[local])
Using cached pydotplus-2.0.2-py3-none-any.whl
Collecting adage>=0.10.1 (from recast-atlas[local])
Using cached adage-0.11.0-py3-none-any.whl (18 kB)
Collecting packtivity>=0.14.23 (from recast-atlas[local])
Using cached packtivity-0.16.2-py3-none-any.whl.metadata (5.8 kB)
Collecting pydot (from recast-atlas[local])
Using cached pydot-1.4.2-py2.py3-none-any.whl (21 kB)
Collecting pygraphviz (from recast-atlas[local])
Using cached pygraphviz-1.11-cp311-cp311-macosx_11_0_arm64.whl
Collecting pyparsing>=2.0.1 (from pydotplus==2.0.2->recast-atlas[local])
Using cached pyparsing-3.1.1-py3-none-any.whl.metadata (5.1 kB)
Collecting psutil (from yadage<0.21.0)
Using cached psutil-5.9.6-cp38-abi3-macosx_11_0_arm64.whl.metadata (21 kB)
Collecting requests>=2.9 (from requests[security]>=2.9->yadage<0.21.0)
Using cached requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting jsonref (from yadage<0.21.0)
Using cached jsonref-1.1.0-py3-none-any.whl (9.4 kB)
Collecting jsonpointer>=1.10 (from yadage<0.21.0)
Using cached jsonpointer-2.4-py2.py3-none-any.whl.metadata (2.5 kB)
Collecting jsonpath-rw (from yadage<0.21.0)
Using cached jsonpath_rw-1.4.0-py3-none-any.whl
Collecting checksumdir (from yadage<0.21.0)
Using cached checksumdir-1.2.0-py3-none-any.whl (5.3 kB)
Collecting glob2 (from yadage<0.21.0)
Using cached glob2-0.7-py2.py3-none-any.whl
Collecting jq (from yadage<0.21.0)
Using cached jq-1.6.0-cp311-cp311-macosx_11_0_arm64.whl.metadata (6.8 kB)
Collecting networkx>=2.4 (from adage>=0.10.1->recast-atlas[local])
Using cached networkx-3.2.1-py3-none-any.whl.metadata (5.2 kB)
Collecting attrs>=22.2.0 (from jsonschema>=3.0.0->recast-atlas[local])
Using cached attrs-23.1.0-py3-none-any.whl (61 kB)
Collecting jsonschema-specifications>=2023.03.6 (from jsonschema>=3.0.0->recast-atlas[local])
Using cached jsonschema_specifications-2023.7.1-py3-none-any.whl.metadata (2.8 kB)
Collecting referencing>=0.28.4 (from jsonschema>=3.0.0->recast-atlas[local])
Using cached referencing-0.30.2-py3-none-any.whl.metadata (2.6 kB)
Collecting rpds-py>=0.7.1 (from jsonschema>=3.0.0->recast-atlas[local])
Using cached rpds_py-0.12.0-cp311-cp311-macosx_11_0_arm64.whl.metadata (3.7 kB)
Collecting mock (from packtivity>=0.14.23->recast-atlas[local])
Using cached mock-5.1.0-py3-none-any.whl.metadata (3.0 kB)
Collecting charset-normalizer<4,>=2 (from requests>=2.9->requests[security]>=2.9->yadage<0.21.0)
Using cached charset_normalizer-3.3.2-cp311-cp311-macosx_11_0_arm64.whl.metadata (33 kB)
Collecting idna<4,>=2.5 (from requests>=2.9->requests[security]>=2.9->yadage<0.21.0)
Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting urllib3<3,>=1.21.1 (from requests>=2.9->requests[security]>=2.9->yadage<0.21.0)
Using cached urllib3-2.0.7-py3-none-any.whl.metadata (6.6 kB)
Collecting certifi>=2017.4.17 (from requests>=2.9->requests[security]>=2.9->yadage<0.21.0)
Using cached certifi-2023.7.22-py3-none-any.whl.metadata (2.2 kB)
Collecting jsonschema>=3.0.0 (from recast-atlas[local])
Using cached jsonschema-4.9.1-py3-none-any.whl (79 kB)
Collecting six>=1.4.0 (from yadage-schemas>=0.10.7->recast-atlas[local])
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 (from jsonschema>=3.0.0->recast-atlas[local])
Using cached pyrsistent-0.20.0-cp311-cp311-macosx_10_9_universal2.whl.metadata (27 kB)
Collecting ply (from jsonpath-rw->yadage<0.21.0)
Using cached ply-3.11-py2.py3-none-any.whl (49 kB)
Collecting decorator (from jsonpath-rw->yadage<0.21.0)
Using cached decorator-5.1.1-py3-none-any.whl (9.1 kB)
Using cached click-8.1.7-py3-none-any.whl (97 kB)
Using cached jsonpointer-2.4-py2.py3-none-any.whl (7.8 kB)
Using cached packtivity-0.16.2-py3-none-any.whl (36 kB)
Using cached jq-1.6.0-cp311-cp311-macosx_11_0_arm64.whl (412 kB)
Using cached PyYAML-6.0.1-cp311-cp311-macosx_11_0_arm64.whl (167 kB)
Using cached requests-2.31.0-py3-none-any.whl (62 kB)
Using cached psutil-5.9.6-cp38-abi3-macosx_11_0_arm64.whl (246 kB)
Using cached recast_atlas-0.3.0-py3-none-any.whl (30 kB)
Using cached certifi-2023.7.22-py3-none-any.whl (158 kB)
Using cached charset_normalizer-3.3.2-cp311-cp311-macosx_11_0_arm64.whl (118 kB)
Using cached networkx-3.2.1-py3-none-any.whl (1.6 MB)
Using cached pyparsing-3.1.1-py3-none-any.whl (103 kB)
Using cached pyrsistent-0.20.0-cp311-cp311-macosx_10_9_universal2.whl (83 kB)
Using cached urllib3-2.0.7-py3-none-any.whl (124 kB)
Using cached mock-5.1.0-py3-none-any.whl (30 kB)
Installing collected packages: ply, glob2, urllib3, six, pyyaml, pyrsistent, pyparsing, pygraphviz, psutil, networkx, mock, jsonref, jsonpointer, jq, idna, decorator, click, checksumdir, charset-normalizer, certifi, attrs, requests, pydotplus, pydot, jsonschema, jsonpath-rw, adage, yadage-schemas, recast-atlas, packtivity, yadage
Successfully installed adage-0.11.0 attrs-23.1.0 certifi-2023.7.22 charset-normalizer-3.3.2 checksumdir-1.2.0 click-8.1.7 decorator-5.1.1 glob2-0.7 idna-3.4 jq-1.6.0 jsonpath-rw-1.4.0 jsonpointer-2.4 jsonref-1.1.0 jsonschema-4.9.1 mock-5.1.0 networkx-3.2.1 packtivity-0.16.2 ply-3.11 psutil-5.9.6 pydot-1.4.2 pydotplus-2.0.2 pygraphviz-1.11 pyparsing-3.1.1 pyrsistent-0.20.0 pyyaml-6.0.1 recast-atlas-0.3.0 requests-2.31.0 six-1.16.0 urllib3-2.0.7 yadage-0.20.1 yadage-schemas-0.11.0Recast Run
(recast_118) ➜ issue_118: recast run examples/rome --backend local --tag debug
2023-11-07 09:47:51,219 | packtivity.asyncback | INFO | configured pool size to 10
2023-11-07 09:47:51,413 | yadage.creators | INFO | initializing workflow with initdata: {'did': 404958, 'dxaod_file': 'https://recastwww.web.cern.ch/recastwww/data/reana-recast-demo/mc15_13TeV.123456.cap_recast_demo_signal_one.root', 'xsec_in_pb': 0.00122} discover: True relative: True
2023-11-07 09:47:51,413 | adage.pollingexec | INFO | preparing adage coroutine.
2023-11-07 09:47:51,413 | adage | INFO | starting state loop.
2023-11-07 09:47:51,430 | yadage.wflowview | INFO | added </init:0|defined|unknown>
2023-11-07 09:47:51,517 | yadage.wflowview | INFO | added </eventselection:0|defined|unknown>
2023-11-07 09:47:51,579 | yadage.wflowview | INFO | added </statanalysis:0|defined|unknown>
2023-11-07 09:47:51,642 | adage.pollingexec | INFO | submitting nodes [</init:0|defined|known>]
2023-11-07 09:47:51,662 | pack.init.step | INFO | publishing data: <TypedLeafs: {'did': 404958, 'dxaod_file': 'https://recastwww.web.cern.ch/recastwww/data/reana-recast-demo/mc15_13TeV.123456.cap_recast_demo_signal_one.root', 'xsec_in_pb': 0.00122}>
2023-11-07 09:47:51,662 | adage | INFO | unsubmittable: 0 | submitted: 0 | successful: 0 | failed: 0 | total: 3 | open rules: 0 | applied rules: 3
2023-11-07 09:47:51,754 | adage.node | INFO | node ready </init:0|success|known>
2023-11-07 09:47:51,754 | adage.pollingexec | INFO | submitting nodes [</eventselection:0|defined|known>]
2023-11-07 09:47:51,755 | pack.eventselection. | INFO | starting file logging for topic: step
2023-11-07 09:48:06,228 | adage.node | INFO | node ready </eventselection:0|success|known>
2023-11-07 09:48:06,228 | adage.pollingexec | INFO | submitting nodes [</statanalysis:0|defined|known>]
2023-11-07 09:48:06,229 | pack.statanalysis.st | INFO | starting file logging for topic: step
2023-11-07 09:48:14,566 | adage.node | INFO | node ready </statanalysis:0|success|known>
2023-11-07 09:48:14,586 | adage.controllerutil | INFO | no nodes can be run anymore and no rules are applicable
2023-11-07 09:48:14,586 | adage.controllerutil | INFO | no nodes can be run anymore and no rules are applicable
2023-11-07 09:48:14,586 | adage | INFO | unsubmittable: 0 | submitted: 0 | successful: 3 | failed: 0 | total: 3 | open rules: 0 | applied rules: 3
2023-11-07 09:48:21,046 | adage | INFO | adage state loop done.
2023-11-07 09:48:21,046 | adage | INFO | execution valid. (in terms of execution order)
2023-11-07 09:48:21,046 | adage | INFO | workflow completed successfully.
2023-11-07 09:48:21,046 | yadage.steering_api | INFO | done. dumping workflow to disk.
2023-11-07 09:48:21,047 | yadage.steering_api | INFO | visualizing workflow.
2023-11-07 09:48:21,364 | recastatlas.subcomma | INFO | RECAST run finished.
RECAST result examples/rome recast-debug:
--------------
- name: CLs 95% based upper limit on poi
value:
exp: 0.8924846399964371
exp_m1: 0.6377501820447065
exp_m2: 0.4731739008380644
exp_p1: 1.2720762961732819
exp_p2: 1.7545752712294322
obs: 1.3352971254860764
- name: CLs 95% at nominal poi
value:
exp: 0.25999040745937085
exp_m1: 0.10547655600578199
exp_m2: 0.03889040527686523
exp_p1: 0.5345040672498215
exp_p2: 0.8276574946063575
obs: 0.574709475331039Logs
(recast_118) ➜ issue_118: grep "docker run" recast-debug/eventselection/_packtivity/eventselection.step.log
docker run --rm -i --cidfile /Users/dnoll/tmp/recast-atlas/issue_118/recast-debug/eventselection/_packtivity/eventselection.cid -v /Users/dnoll/tmp/recast-atlas/issue_118/recast-debug/eventselection:/Users/dnoll/tmp/recast-atlas/issue_118/recast-debug/eventselection:rw -v /Users/dnoll/tmp/recast-atlas/issue_118/recast-debug/init:/Users/dnoll/tmp/recast-atlas/issue_118/recast-debug/init:rw docker.io/reanahub/reana-demo-atlas-recast-eventselection:1.0 sh -c bash
2023-11-07 09:48:06,168 | pack.eventselection. | DEBUG | moving on from subprocess execution: docker run --rm -i --cidfile /Users/dnoll/tmp/recast-atlas/issue_118/recast-debug/eventselection/_packtivity/eventselection.cid -v /Users/dnoll/tmp/recast-atlas/issue_118/recast-debug/eventselection:/Users/dnoll/tmp/recast-atlas/issue_118/recast-debug/eventselection:rw -v /Users/dnoll/tmp/recast-atlas/issue_118/recast-debug/init:/Users/dnoll/tmp/recast-atlas/issue_118/recast-debug/init:rw docker.io/reanahub/reana-demo-atlas-recast-eventselection:1.0 sh -c bash
2023-11-07 09:48:06,168 | pack.eventselection. | DEBUG | finally for docker run --rm -i --cidfile /Users/dnoll/tmp/recast-atlas/issue_118/recast-debug/eventselection/_packtivity/eventselection.cid -v /Users/dnoll/tmp/recast-atlas/issue_118/recast-debug/eventselection:/Users/dnoll/tmp/recast-atlas/issue_118/recast-debug/eventselection:rw -v /Users/dnoll/tmp/recast-atlas/issue_118/recast-debug/init:/Users/dnoll/tmp/recast-atlas/issue_118/recast-debug/init:rw docker.io/reanahub/reana-demo-atlas-recast-eventselection:1.0 sh -c bash
(recast_118) ➜ issue_118: echo $PACKTIVITY_DOCKER_CMD_MOD
(recast_118) ➜ issue_118: If you are interested we could troubleshoot why you need the PACKTIVITY_DOCKER_CMD_MOD and I don't.
But we do not necessarily need to do this and I am also happy to just leave things as they are.
Also if we troubleshoot this we should probably move the discussion to a new issue.
This issue is correctly "connected to" and "resolved with" yadage/yadage#116.
from recast-atlas.
matthewfeickert
commented on June 16, 2024
If you are interested we could troubleshoot why you need the
PACKTIVITY_DOCKER_CMD_MODand I don't.
But we do not necessarily need to do this and I am also happy to just leave things as they are.
That might be interesting, but I suspect that this would be related to id groups on our local machines. I'll open up an Issue though for the future as having as unified of an experience is beneficial in the long run. 👍
Though the first order of priority when I get time to do recast development again is fixing yadage/yadage#116 and then getting that tested with the REANA team.
from recast-atlas.
Related Issues (20)
- how to assign cpu cores and memory limits to a job when running via docker as backend HOT 2
- `local` and `reana` extras running into PyYAML 5.4.x Cython issues HOT 3
- Recast on ARM Architectures HOT 4
- Unify UX for 'local' backend
- lxplus apptainer can't switch between registries HOT 2
- Add ability to set platform from packtivity environmental variable
- test_run_hello_world test fails on Python 3.9 HOT 2
- Workflow visualization shows only square glyphs in place of titles since v0.1.1 HOT 15
- eval "$(recast auth reana-setup)" fails HOT 4
- missing coverage for a few important ccommands
- require 'jsonschema<4.0' HOT 3
- `recast backends ls --check` should check only the installed backends
- `yadageschemas` fails to specify `six` dependency HOT 2
- Add ability to list out environmental variables used for configuration
- LXPLUS example in docs not working HOT 3
- Remove deprecated distutils functionality
- ATLAS-CONF-2018-041 example fails HOT 1
- Ability to avoid pull step to use local image HOT 1
- Add a HEAD of dependencies CI workflow
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from recast-atlas.