Code Monkey home page Code Monkey logo

panama's Introduction

,-.----.                           ,--.das     nd               ____ ulticore utils for corsik  7
\    /  \     ,---,              ,--.'|   ,---,               ,'  , `.                    ,---,
|   :    \   '  .' \         ,--,:  : |  '  .' \           ,-+-,.' _ |                   '  .' \
|   |  .\ : /  ;    '.    ,`--.'`|  ' : /  ;    '.      ,-+-. ;   , ||                  /  ;    '.
.   :  |: |:  :       \   |   :  :  | |:  :       \    ,--.'|'   |  ;|                 :  :       \
|   |   \ ::  |   /\   \  :   |   \ | ::  |   /\   \  |   |  ,', |  ':                 :  |   /\   \
|   : .   /|  :  ' ;.   : |   : '  '; ||  :  ' ;.   : |   | /  | |  ||                 |  :  ' ;.   :
;   | |`-' |  |  ;/  \   \'   ' ;.    ;|  |  ;/  \   \'   | :  | :  |,                 |  |  ;/  \   \
|   | ;    '  :  | \  \ ,'|   | | \   |'  :  | \  \ ,';   . |  ; |--'                  '  :  | \  \ ,'
:   ' |    |  |  '  '--'  '   : |  ; .'|  |  '  '--'  |   : |  | ,                     |  |  '  '--'
:   : :    |  :  :        |   | '`--'  |  :  :        |   : '  |/                      |  :  :
|   | :    |  | ,'        '   : |      |  | ,'        ;   | |`-'                       |  | ,'
`---'.|    `--''          ;   |.'      `--''          |   ;/                           `--''
  `---`                   '---'                       '---'

PANAMA - A python toolkit for CORSIKA7.

Read the Docs

GitHub Workflow Status GitHub issues Codecov

PyPI DOI GitHub Codestyle

Features

This python package provides multiple features -- each feature can be used independently, but they also work great together.

  • Execute CORSIKA7 on multiple cores
  • Read CORSIKA7 DAT files ("particle files") to pandas DataFrames
    • Correctly parse output from the EHIST option
  • Calculate weights for a multiple primary spectra

To see some examples on how to use panama, see the introduction in the documentation. To get an overview of how the features play together, have a look at the example notebook in the documentation. In-depth explanation is provided in the API documentation.

Installation

pip install corsika-panama

If you want to convert Corsikas DAT files to HDF5 files, you need to install the optional hdf dependency

pip install corsika-panama[hdf]

CORSIKA7

For usage and installation of CORSIKA7, please refer to its website and its userguide. To properly use this package, knowledge of CORSIKA7 is required.

If you want to install CORSIKA7, you need to request access to their CORSIKA7 mailing list, as described on their website, then you will receive the CORSIKA7 password. If you want to skip the process of getting familiar with the software and compiling it with coconut, panama provides a (linux) script for compiling it. You will need a fortran compiler. CORSIKA7 will then be pre-configured with the curved earth, EHIST, SIBYLL2.3d and URQDM options. For finer control over the used options, please compile CORSIKA7 yourself. After cloning this repository, you can then execute

CORSIKA_VERSION=77500 CORSIKA_PW=CORSIKA_PASSWORD_YOU_WILL_RECEIVE_BY_MAIL admin/download_corsika.sh

which will download and compile CORSIKA7 version 77500. If you are interested in automatically testing software using CORSIKA7, using GitHub actions, have a look at the .github folder of this project in combination with the admin script.

Contributing

Contributions and suggestions are very welcome. Feel free to open an issue or pull request. This project uses pdm for the build system as well as a dependency and virtual environment manager. For suggestions on how to set up a development environment, have a look at CONTRIBUTING.md.

Further Notes

This project tries to stay compatible with the suggestions from Scikit hep.

Naming idea goes back to @Jean1995, thanks for that! He originally proposed "PArallel ruN of corsikA on MAny cores", as the scope of this library grew bigger, it evolved into the current name.

This started as part of the code I wrote for my master thesis. I ended in the same place where most CORSIKA7 users end when running large CORSIKA7 simulations and wrote small scripts to split one simulation request into multiple CORSIKA7 processes with different seeds. The FACT software (fact-project/corsika_wrapper) and the IceCube software does essentially the same thing (and I am sure, MAGIC, CTA and other air-shower based observatories do the same). I hope this package provides a more experiment-independent and better documented version of internal software packages.

Related Repositories

panama's People

Contributors

dependabot[bot] avatar jean1995 avatar pre-commit-ci[bot] avatar the-ludwig avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

Forkers

jean1995

panama's Issues

stop copying the whole run folder

It should be possible to "trick" CORSIKA7. Symlinking most files in the run directory should be enough. For example the IceCube runner does this.
This would dramaticly increase startup time. Most of it is wasted to copying CORSIKA7

Add support for `tox`

The scikit-hep team suggests using tox to help new collaborators get started. See this tutorial.

Appart from that, we already saw multiple bugs related to the python version, so this should be used anyway.

Installaton not possible on arm architecture

I am unable to install PANAMA on my M1 MacOS, it fails with the error

  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [23 lines of output]
      Error in sitecustomize; set PYTHONVERBOSE for traceback:
      AssertionError:
      cpuinfo failed, assuming no CPU features: 'flags'
      * Using Python 3.11.1 (main, Dec 23 2022, 09:28:24) [Clang 14.0.0 (clang-1400.0.29.202)]
      Traceback (most recent call last):
        File "/opt/homebrew/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
          main()
        File "/opt/homebrew/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/homebrew/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/opt/homebrew/Cellar/[email protected]/3.11.1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/homebrew/Cellar/[email protected]/3.11.1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
          self.run_setup()
        File "/opt/homebrew/Cellar/[email protected]/3.11.1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/build_meta.py", line 335, in run_setup
          exec(code, locals())
        File "<string>", line 265, in <module>
        File "<string>", line 132, in get_blosc2_directories
      NotADirectoryError: Install directory for blosc2 not found in /opt/homebrew/Cellar/[email protected]/3.11.1/Frameworks/Python.framework/Versions/3.11
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Might be related to this issue

Road to 1.0

I am working on the very last issues and I think after updating the documentation, this package is ready for a 1.0 release.

I have a lot of ideas for this package which might improve and extend it, but I think the current scope is very reasonable.
For a 2.0 release some features may be:

  • panama install 77420 asks for the corsika password and downloads and installs/compiles the given corsika version to a given location (this would be a modularization and extention of the script in the admin folder. But may be we don't want to steal coconut the show)
  • yaml steering: instead of steering cards, write one steering card where everything is replaced by a template, which is then filled in by values from a yaml file. optionally override this on the command line.

@Jean1995 as you and I are currently the only ones using this, I guess (maybe @pgutjahr?): Are there any more features you might want to see for a 1.0?

maximal job mode

Currently, panama run --jobs N starts N jobs per primary particle. So if we have 5 components, we potentially have 5*N particles.
On a multi-user system we want to be able to limit the number of total jobs we have. So we need to implement a --max-job option to run the components after each other, not all at the same time.
While at it, a class-based refactoring of run.py is needed. See branch max-threads

`read_DAT` broken for newest pandas release

I'm trying to read files with the read_DAT option, my command is

run_header, _, particles_C7 = pn.read_DAT(glob=f"{C7_PATH}/DAT*[!.long]", mother_columns=True, additional_columns=True, max_events=MAX_EVENTS_C7)

If I use the most recent version of pandas (2.1.1), I get an error:

  File "/home/xxxxx/xxxxx/xxxxx/create_history_plots.py", line 36, in <module>
    run_header, _, particles_C7 = pn.read_DAT(glob=f"{C7_PATH}/DAT*[!.long]", mother_columns=True, additional_columns=True, max_events=MAX_EVENTS_C7)
  File "/usr/local/lib/python3.10/dist-packages/panama/read.py", line 306, in read_DAT
    add_mother_columns(df_particles, pdgids)
  File "/usr/local/lib/python3.10/dist-packages/panama/read.py", line 360, in add_mother_columns
    df_particles.loc[~df_particles["has_mother"].array, "mother_hadr_gen"] = pd.NA
  File "/usr/local/lib/python3.10/dist-packages/pandas/core/indexing.py", line 881, in __setitem__
    indexer = self._get_setitem_indexer(key)
  File "/usr/local/lib/python3.10/dist-packages/pandas/core/indexing.py", line 758, in _get_setitem_indexer
    return self._convert_tuple(key)
  File "/usr/local/lib/python3.10/dist-packages/pandas/core/indexing.py", line 963, in _convert_tuple
    keyidx = [self._convert_to_indexer(k, axis=i) for i, k in enumerate(key)]
  File "/usr/local/lib/python3.10/dist-packages/pandas/core/indexing.py", line 963, in <listcomp>
    keyidx = [self._convert_to_indexer(k, axis=i) for i, k in enumerate(key)]
  File "/usr/local/lib/python3.10/dist-packages/pandas/core/indexing.py", line 1484, in _convert_to_indexer
    return self._get_listlike_indexer(key, axis)[1]
  File "/usr/local/lib/python3.10/dist-packages/pandas/core/indexing.py", line 1520, in _get_listlike_indexer
    keyarr, indexer = ax._get_indexer_strict(key, axis_name)
  File "/usr/local/lib/python3.10/dist-packages/pandas/core/indexes/base.py", line 6114, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "/usr/local/lib/python3.10/dist-packages/pandas/core/indexes/base.py", line 6175, in _raise_if_missing
    raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index([ True,  True,  True,  True,  True,  True,  True,  True,  True,  True,\n       ...\n       False,  True,  True,  True, False,  True,  True,  True, False,  True],\n      dtype='bool', length=1119605)] are in the [index]"

However, this error vanished when I used pandas 2.0.3 before, so something in pandas must have changed which broke the script.

switch to pdm

Upon further research and experience, I no longer think poetry is the build system to use for this project. pdm seems far better suited. Migration should not take too long.

HDF5 installation not found Mac M1 Pro Ventura

The installation using pip install corsika-panama fails on a Mac M1 Pro with Ventura 13.6.1 because the local installation of HDF5 is not found.

This can be fixed by setting the correct installation path to
export HDF5_DIR=/opt/homebrew/opt/hdf5

which of course requires the installation of HDF5 using brew install hdf5.

Save STDOUT

Right now, the stdout of the CORSIKA files is not saved, this should be added

`get_version()` broken?

I've tried installing version 0.6.0 of panama, both on a Linux cluster and my local MacOS, and I get the error messages:

Traceback (most recent call last):
  File "/usr/local/bin/panama", line 5, in <module>
    from panama.cli import cli
  File "/usr/local/lib/python3.10/dist-packages/panama/__init__.py", line 19, in <module>
    __version__ = get_version()
  File "/usr/local/lib/python3.10/dist-packages/panama/__init__.py", line 14, in get_version
    with open(path) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.10/dist-packages/pyproject.toml'

for Linux and

Traceback (most recent call last):
  File "/opt/homebrew/bin/panama", line 5, in <module>
    from panama.cli import cli
  File "/opt/homebrew/lib/python3.11/site-packages/panama/__init__.py", line 19, in <module>
    __version__ = get_version()
                  ^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/panama/__init__.py", line 14, in get_version
    with open(path) as f:
         ^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/opt/homebrew/lib/python3.11/site-packages/pyproject.toml'

for MacOS.

Start with different EVTNR

I believe at the moment, the first event always starts with the event number 1

It would be nice if one could optionally specific a different start number, e.g., if one wants to organize the events in a specific way :)

Update documentation

The documentation is heavily out of date. Especially the examples from the readme need to be updated

fix test protons_only

I test the weighting and corsika reading via the following mechanisms:

  • fit the weighted primary spectrum -> spectral index should be around 2.7
  • fit the weighted muon spectrum of conventional muons -> spectral index should be around 3.7
  • fit the weighted muon spectrum of prompt muons -> spectral index should be around 2.7

I do this for weighting with the HillasGaisserH3a model, which works and I do this with the Thunman model. This does not work, since thunman only has protons as primaries, and then we have a really low statistic. This should be fixed

Move panama/fluxes to own package

I implemented numpy/pandas compatible versions of multiple cr-flux models for this package.
This is not really what this package is designed to do (PANdas And Multicore utils for corsikA7) and could be useful also outside this package.

use corsikaio/noparse

@maxnoe created the branch corsikaio/noparse in pycorsikaio for me.
Implementing that should dramatically increase loading speeds.

mother readout information wrong

#80 introduced a huge bug where the information from the mother columns is not read out correctly.
(Since pandas is indexing with the dataframe index, where I intend to index with the integer index.)
All previous versions work fine.

Working on the fix right now, also adding a test case to catch this.

fix test of created files

In tests/files/compare there are the files I produced by running the test_run script on my local machines. I provide them in the hope, that we can corss-check if the corsika runner produces the same files on different machines, when using the same seed.
This, sadly, does not seem to be the case

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.