cta-observatory / cta-benchmarks Goto Github PK

View Code? Open in Web Editor NEW

4.0 8.0 5.0 9.57 MB

Collection of benchmarking code for cta

Home Page: https://cta-observatory.github.io/cta-benchmarks/

License: MIT License

Jupyter Notebook 54.61% Python 42.45% Shell 2.94%

benchmarks papermill notebook cta

cta-benchmarks's Introduction

cta-benchmarks

Collection of benchmarking code for cta

Note that the benchmarks require some input data that is not provided in the repository. No data files should be included in this repo to avoid causing its size to increase rapidly instead, raw data files are provided on a dedicated server.

General Structure

Preparation: notebooks that generate testing data (e.g. process raw data and produce data products that are needed as input for various benchmarks)
Benchmarks: notebooks that perform a certain benchmark. These should produce a final plot or value that can be recorded using the storage functionality of papermill
Summaries: notebooks that collect the final results from multiple Benchmarks and present an executive summary of the results.

Guidelines

General:

Do not commit notebooks to this repo that contain output (please strip the output first). This makes the repo size stay small and makes looking at differences much easier. The outputs will eventually be included automatically in the documentation, not in the repo.
Do not commit input or output data files to this repo (see above)
separate the production of benchmark data (e.g. DL1, DL2, or DL3 files) from the benchmarks themselves. The production of data files should be in notebooks in the Preparation/ directory (data prep).

From this Netflix post:

Low Branching Factor: Keep your notebooks fairly linear. If you have many conditionals or potential execution paths, it becomes hard to ensure end-to-end tests are covering the desired use cases well.
Library Functions in Libraries: If you do end up with complex functions which you might reuse or refactor independently, these are good candidates for a coding library rather than in a notebook. Providing your notebooks in git repositories means you can position shared unit-tested code in that same repository as your notebooks, rather than trying to unit test complex notebooks.
Short and Simple is Better: A notebook which generates lots of useful outputs and visuals with a few simple cells is better than a ten page manual. This makes your notebooks more shareable, understandable, and maintainable.

Submit a benchmark

You may open an issue to discuss the benchmark you want to create before
Add your notebook to the relevant folder
Use the standard input parameters (see below)
Strip out the outputs : nbstripout your-awesome-benchmark.ipynb
Check that your notebook in running well with papermill your-awesome-benchmark.ipynb awesome-output.ipynb -f yourparameterfile.yml and check that awesome-output.ipynb looks like what you expect
Make a pull request

Note: create yourparameterfile.yml by copying and modifying config/parameters_jenkins.yml to your local data paths.

Parameterise your notebook

In cta-benchmarks we use papermill to run parameterized notebooks.
When you propose a benchmark, put all the parameters in one unique cell and tag it as parameters (see the papermill page).
The parameters are global and common to all benchmarks.
Especially:

the path to raw data directory is given by input_dir
the path to the prepared data (output of the notebooks in Preparation) is given by prepared_data_dir

Available data on the running server

(see config/parameters_jenkins.yml)

gamma_diffuse: 'gamma_40deg_0deg_run102___cta-prod3-lapalma3-2147m-LaPalma_cone10.simtel.gz'

Setup for automatic running of all benchmarks:

This software uses the Anaconda python distribution, which must be installed first.

1. Create benchmark environment:

conda env create --file environment.yml

This will install ctapipe and all dependencies needed to run the benchmarks. This only needs to be done once.

2. Change to the benchmark environment

conda activate cta-benchmarks

This must be done every time you open a new terminal and want to run the benchmarks.

3. Run the benchmarks:

python build.py

This will run all notebooks in each of the following directories in order:

Preparation/
Benchmarks/
Summaries/

The output will be in a directory called BUILD/

You can of course run each benchmark individually using jupyter notebook or jupyter lab (the latter requires that you run conda install jupyter-lab first)

cta-benchmarks's People

Contributors

Stargazers

Watchers

Forkers

vuillaut kosack thomasarmstrong moralejo healthypear

cta-benchmarks's Issues

Use pytables instead of h5py

Until the code can rely on proper ctapipe event source for DL>1 data levels,
pytables is recommended over h5py to read data.

Suggestion: Manage notebooks in common way

I love the idea of benchmarks being a set of Jupyter notebooks, so they are easier to add and read.

The problem with notebooks in GIT is always:

git diffs don't mean much, especially if output is in the notebook
small changes change metadata, etc
code style is hard to maintain

There is a nice solution from the GammaPy developers: their gammapy jupyter command lets you manage a set of notebooks in a common way:

gammapy jupyter strip : removes output, so diffs make sense (use nbdime diff for even better diffs)
gammapy jupyter black runs black code formatter on cells in all notebooks

etc.

Then they only commit stripped noteboooks to git, and automatically run the notebooks producing documentation in Sphinx format for viewing.

Can we re-use this technique? Or perhaps ask them to split out that functionality into a stand-alone tool, so gammapy is not needed?

Integration Dispersion or Charge Resolution?

Is there a reason the "integration dispersion" is used as opposed to the Charge Resolution?

For those not aware: the Charge Resolution compares the measured (integrated + calibrated) charge to the true charge. Analogous to the Root-Mean-Square Error, the fractional Charge Resolution sigma_q/Q_T for a particular “true charge” Q_T (the number of photoelectrons that were produced in the sensor, before multiplication) is defined as:

where N is the total number of measured charges, Q_M_i , with that value of Q_T. It therefore takes into account both the bias and the spread of the measured charge about the "true charge".

When using the photo_electron_image from the simtelarray files as Q_T, the Poisson fluctuations in photoelectron number need to be included into the equation:

This ensures that a perfect detector which consistently reads-out a “measured charge” with an equal value to the “true charge” would still hit the Poisson limit. This limit ensures realistic conclusions can be reached from the Monte Carlo simulations, as it is not physically possible to know the “true charge” generated inside the photosensor, free from fluctuations.

An example of a Charge Resolution plot, comparing a result of a simulated camera at different values of NSB, is shown below:

In the past, the Charge Resolution had CTA Requirement attached to it, however it has since been updated/replaced with the "Intensity Resolution", where the measured signal in photons is compared to the true signal in photons. This change of definition from photoelectrons to photons was performed on the majority of the requirements, as it is a more coherent representation of performance in observing Cherenkov showers, especially with the variety of cameras and photosensors in CTA.

To expand on that, in the case of SiPMs it is possible to reduce the bias voltage applied to reduce the Excess Noise Factor, therefore improving the Charge Resolution. However, this also reduces the Photon Detection Efficiency, which is also important to consider. This is correctly reflected in the Intensity Resolution, therefore making it a better requirement than Charge Resolution.

The definition of the Intensity Resolution can be found on Jama:

As explained, the full proof of a camera meeting this requirement is performed with MC simulations of Cherenkov showers, the Charge/Intensity Resolution of the real camera in the lab, and the proof that the model matches reality with the comparison against the MC simulation of the camera in the lab. It is also important to note that the requirement must be met under specified levels of NSB.

How we should obtain Intensity Resolution from MC is currently unclear - the true number of photons is not stored by default in the simtelarray file. One approximation is to multiply the number of photoelectrons by the Quantum Efficiency/Photon Detection Efficiency.

For the time being, the Charge Resolution can be calculated in ctapipe using the ctapipe.analysis.camera.charge_resolution.ChargeResolutionCalculator class. A script which uses this class is located in ctapipe/tools/extract_charge_resolution.py. It produces a HDF5 file containing the Charge Resolution at each true charge.

There was a suggestion in cta-observatory/ctapipe/pull/857 to create a extract_dl1 Tool, which creates a file containing the DL1 containers, which then can be read into scripts like extract_charge_resolution.py. This will simplify the script and not require it to loop over and calibrate events itself.

The class ctapipe.plotting.charge_resolution.ChargeResolutionPlotter can plot multiple Charge Resolution results onto the same axis. It also contains the equations for the old Charge Resolution requirements, which can also be plot onto the axis. A script which uses this class can be found at ctapipe/tools/plot_charge_resolution.py, which can take a list of files produced from extract_charge_resolution, and plot them on the same axis.

There are many factors which contribute to the Charge/Intensity resolution of a camera. It encapsulates the electronics performance, the waveform calibration (R0->DL0), the signal extraction approach, and the signal calibration. Additionally, the answer to "which charge integration is best for my camera?" can also depend on "what NSB are you observing at?" and "are you illuminating the camera with Cherenkov Shower ellipses or a uniform illumination?". It is therefore appropriate to use the Charge/Intensity Resolution as a benchmark for the charge extraction, as it has associated requirements that the camera must meet anyway, of which define the conditions the benchmark must be assessed under.