Code Monkey home page Code Monkey logo

framework's Introduction

The REST Framework

DOI pipeline status website api forum

The REST-for-Physics (Rare Event Searches Toolkit) Framework is mainly written in C++ and it is fully integrated with ROOT I/O interface. REST was initially born as a collaborative software effort to provide common tools for acquisition, simulation, and data analysis of gaseous Time Projection Chambers (TPCs). However, the framework is already extending its usage to be non-exclusive of detector data analysis. The possibilities of the framework are provided by the different libraries and packages written for REST in our community.

The REST Framework provides 3 interfaces that prototype the use of event types, metadata and event processes through TRestEvent, TRestMetadata and TRestEventProcess abstract class definitions. Any REST library will implement specific objects that inherit from those 3 basic interfaces.

Different event processes can be combined to build complex event processing chains with full traceability. The metadata objects will allow us to provide input parameters or information to the framework using a XML-like format. REST integrates a special metadata object named TRestManager that encapsulates all the required information to launch the processing of a particular data chain. REST will produce output using ROOT format. Any REST file will always contain a TRestRun metadata object. TRestRun is a metadata object responsible to encapsulate and give access to all the objects stored inside the REST/ROOT file; i.e. the specific resulting TRestEvent output, the TRestAnalysisTree, and any specific TRestMetadata object used during a processing chain.

This framework provides additionally different interfaces to browse data, TRestBrowser, event visualization TRestEventViewer, define a event data processing infraestructure, TRestProcessRunner, event analysis and metadata plotting, TRestAnalysisPlot or TRestMetadataPlot, a common access analysis tree based on TTree ROOT object, TRestAnalysisTree, and centralizing the use of REST through a manager TRestManager are few of the features the framework offers when used standalone.

Other objects included in the framework will help to add unit definitions, REST_Units, define physical constants and basic physical routines, REST_Physics or access to geometrical calculations, TRestMesh. Additional objects provide methods to help on text formatting as TRestStringHelper or define output styles, TRestStringOutput.

Basic pure analysis tasks will also be included in this framework, such as a processes performing fundamental routines, such as performing generic fits on observables/branches found inside the analysis tree, producing a summary report, creating data quality rules definitions, or basic interfaces to external databases.

Mirror repositories

REST is mirrored to the following repositories where pipelines are executed, and where code can also be retrieved.

Code can be pulled for read-access from those mirrors, however, development is centralized at the main GitHub public repository.

Installing

Please, visit the REST-for-Physics userguide for installation instructions.

Useful links or references

Contributing

Please read CONTRIBUTING.md to get some guidelines on how to contribute to this project. Before any contribution, those guidelines must be assimilated and accepted. In any case, changes, improvements, or addons, to CONTRIBUTING.md are aceptable after proposal and discussion with other authors at the REST Framework forum.

The framework exploits the Git tagging system to produce its own versioning system. It is important to emphasize that the REST framework centralizes the versioning of all the submodules (libraries, packages, ...) that it contains. Details on how the REST version number is produced are given in CONTRIBUTING.md.

Versioning

Any metadata object written with REST will be stamped with few metadata members that will allow to identify the state of the code when the object was produced. Those data members are:

  • fVersion: A string containing the human version number.
  • fCommit: The latest commit hash value when the compilation took place.
  • fLibraryVersion: The human version library. It is fixed by CMakeLists at the library submodules.
  • fOfficialRelease: It will be true if the commit was tagged at the repository.
  • fCleanState: It will be true if there are no local modifications (including submodules). To remove any local modifications and recover a clean state we may execute source clean-state.sh at the project root.

If different REST versions were used to write a ROOT file, e.g. at different steps of the data processing chain, the historic metadata objects will preserve their original version. However, the TRestRun metadata object will always store the version used to write the ROOT file.

After REST release 2.2.1., REST implements correctly the ROOT schema evolution. Therefore, any new REST version should always be backwards compatible. I.e. Any file written after v2.2.1 should be readable without problems with any future version.

A major change at 2.3 will prevent from backwards compatibility, since class names have been reviewed.

Publications

Presentations

License

This project is licensed under the GNU License - see the LICENSE file for details

Acknowledgments

We acknowledge support from the the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme, grant agreement ERC-2017-AdG788781 (IAXO+), and from the Spanish Agencia Estatal de Investigacion under grant FPA2016-76978-C3-1-P

Insitution logos

framework's People

Contributors

alvaroezq avatar andrii01l avatar catofes avatar cmargalejo avatar daviddiezib avatar ddn2 avatar glorialuzon avatar igarciai avatar jgalan avatar johnwu1111 avatar jovoy avatar jporron avatar juanangp avatar kjakov avatar konradaltenmueller avatar lhprojects avatar lobis avatar nkx111 avatar oscar-pl avatar pre-commit-ci[bot] avatar raulena333 avatar vicctorps avatar vindaar avatar xiechensjtu avatar ybedfer avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

framework's Issues

TRestMetadata::GetDblParameterWithUnits issue for scientific notation

When this function is used to extract the value field from an xml file, any entry such as

<parameter name="some_parameter" value=1.23e-4 />

fails because the "e" is interpreted as a unit. As a result, the parameter value in the example above would be interpreted as 1.23.

I believe the problematic part is here:

string RemoveUnitsFromString(string s) { return s.substr(0, s.find_first_not_of("1234567890(),.-")); }

Presentation at the ROOT users workshop 2022

Dear colleagues,

The ROOT team (https://github.com/root-project/root/) is organising a users workshop this year. The format will be fully remote. The tentative dates are May 9th-11th, 3 hours per day starting at 15:00 Geneva time.

We would like to know if any of the contributors to the REST-For-Physics framework would like to do a presentation about it at the workshop, we'd love to hear your feedback on ROOT!

Thank you in advance,

Enric

TRestMesh node data members improved design

The TRestMesh coordinates, energy and group id data members would be better encapsulated into a single structure so that we define for example

struct MeshNode {
UInt_t GroupID;
Int_t X;
Int_t Y;
Int_t Z;
Double_t Energy;
}

Then, we would replace

    /// A vector storing the group ID of the corresponding nodes activated
    std::vector<Int_t> fNodeGroupID;
    /// A vector storing the X-dimension cell id
    std::vector<Int_t> fNodeX;
    /// A vector storing the Y-dimension cell id
    std::vector<Int_t> fNodeY;
    /// A vector storing the Z-dimension cell id
    std::vector<Int_t> fNodeZ;
    /// A vector storing the total energy inside the cell id
    std::vector<Double_t> fEnergy;

by

std::vector <MeshNode> fMeshNodes;

This issue was brought by @juanangp at PR #73

TRestStringOutput sometimes causes crash at the end of a simulation

Edit: Thanks to @nkx111 the issue was found in TRestStringOutput, probably it was related to resizing the console due to using screen. The issue is now solved in commit f8a2e8b.

I found out that if I don't set the runNumber to auto I sometimes get a crash at the end of the simulation. Here are the relevant lines in the output with debug enabled.

DEBUG: Energy deposited in SENSITIVE volume: 0 keV
DEBUG: Events depositing energy in sensitive volume: 36713/10000
DEBUG: End of event ID 9999 (10000 of 10000)

=======================================================================
======       Pre-compound/De-excitation Physics Parameters     ========
=======================================================================
Type of pre-compound inverse x-section              3
Pre-compound model active                           1
Pre-compound low energy (MeV)                       0.1
Type of de-excitation inverse x-section             3
Type of de-excitation factory                       Evaporation
Number of de-excitation channels                    8
Min excitation energy (keV)                         0.01
Min energy per nucleon for multifragmentation (MeV) 1e+05
Level density (1/MeV)                               0.1
Time limit for long lived isomeres (ns)             1442.7
Internal e- conversion flag                         1
Store e- internal conversion data                   1
Electron internal conversion ID                     0
Correlated gamma emission flag                      0
Max 2J for sampling of angular correlations         10
=======================================================================
======================== run summary ======================
10000 Events simulated
===========================================================
Graphics systems deleted.
Visualization Manager deleting...
-- Debug : TRestRun::WriteWithDataBase. Getting entries in analysisTree
-- Debug : TRestRun::WriteWithDataBase. Entries found : 36713
-- Debug : TRestRun::WriteWithDataBase. Calling this->Write(0,kWriteDelete)
-- Debug : TRestRun::WriteWithDataBase. Succeed
-- Debug : TRestRun::WriteWithDataBase. fMetadata.size() == 2
-- Debug : TRestRun::WriteWithDataBase. fInputMetadata.size() == 0
-- Debug : NO historic
-- Debug : TRestRun::WriteWithDataBase. fInputMetadata.size() == 0
-- Debug : NO historic
-- Debug : TResRun::WriteWithDataBase. Run number is : 0
terminate called after throwing an instance of 'std::length_error'
  what():  basic_string::_M_create
CosmicParticles.sh: línea 28: 31047 Abortado                restG4 CosmicParticles.rml

I cannot reproduce this issue though, I tried doing a shorter simulation also setting the runNumber to 0 but this time everything went fine.

Based on the output log I think the problem occurs in line 951 of TRestRun.cxx, in the block

// write to database
   debug << "TResRun::WriteWithDataBase. Run number is : " << fRunNumber << endl;
   if (fRunNumber != -1) {
       int fileid = gDataBase->set_runfile(fRunNumber, (string)fOutputFileName);
       fout << "DataBase Entry Added! Run Number: " << fRunNumber << ", File ID: " << fileid << endl;
   }

Dis-ordered event ids when using multi-thread

The event ids will be dis-ordered when we enabled multi-thread. For example, the event id will be 1,3,4,2,5,6,8,7... at entry 1,2,3,4,5,6,7,8. This is because the thread running time for each event is in-predictable. On the other hand, the overall tendency of event id is still increasing.

We can fix this with certain "buffer" techniques, buy storing output event and observables in a buffer and write only the event with closest event id. This may slow down the speed.

Remove FindROOT.cmake

Currently in the repository /cmake directory we have some Find*.cmake macros that in my opinion should be removed. In particular FindROOT.cmake.

when invoking find_package(ROOT) cmake macro, it will search for the corresponding macro, first in the project directory. If it doesn't find it there, it will search on some other standard directories and find it (if ROOT is installed and sourced). The problem is that since we have our own FindROOT.cmake macro, its overwritting the installation macro. Our macro is probably based on some specific old root version. We should use the current root installation macro.

The fix should be as simple as removing the old macro, however there are some very strange bugs happening (which took my quite a while to relate to this macro...).

In particular when doing the make it will fail in the first line giving:

/bin/sh: 1: -f: not found

I want to get it removed now because I think its causing issues when linking to a newer version of Garfield, throwing errors like:

CMake Error at cmake/MacroRootDict.cmake:509 (add_library):
  Target "RestDetector" links to target "ROOT::Geom" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?
Call Stack (most recent call first):
  source/libraries/detector/CMakeLists.txt:30 (COMPILELIB)


CMake Error at cmake/MacroRootDict.cmake:509 (add_library):
  Target "RestDetector" links to target "ROOT::Gdml" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?
Call Stack (most recent call first):
  source/libraries/detector/CMakeLists.txt:30 (COMPILELIB)


CMake Error at cmake/MacroRootDict.cmake:509 (add_library):
  Target "RestDetector" links to target "ROOT::Graf3d" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?
Call Stack (most recent call first):
  source/libraries/detector/CMakeLists.txt:30 (COMPILELIB)


-- Generating done
CMake Generate step failed.  Build files cannot be regenerated correctly.

[Failed to reload]

I have no idea what we are doing that is causing the -f error, any ideas are welcome.

ROOTtoRML tool

It would be interesting to create a simple tool that reads a ROOT file and generates a RML file. So that if I find a given number of processes at the ROOT file, I place them at the RML and I could process the file exactly as it was processed. Then, of course, I would be able to modify that RML to play with the processing of the data.

restRoot --m 1 produces unexpected results

After the upgrade to the new libraries scheme errors come out when loading the macros into the restRoot environment.

Error in cling::AutoloadingVisitor::InsertIntoAutoloadingState:
   Missing FileEntry for TRestGeant4Particle.h
   requested to autoload type TRestGeant4Particle
Error in cling::AutoloadingVisitor::InsertIntoAutoloadingState:
   Missing FileEntry for TRestGeant4ParticleCollection.h
   requested to autoload type TRestGeant4ParticleCollection
Error in cling::AutoloadingVisitor::InsertIntoAutoloadingState:
   Missing FileEntry for TRestGeant4ParticleSource.h
   requested to autoload type TRestGeant4ParticleSource
Error in cling::AutoloadingVisitor::InsertIntoAutoloadingState:
   Missing FileEntry for TRestGeant4PhysicsLists.h
   requested to autoload type TRestGeant4PhysicsLists
Error in cling::AutoloadingVisitor::InsertIntoAutoloadingState:
   Missing FileEntry for TRestGeant4PrimaryGenerator.h
   requested to autoload type TRestGeant4PrimaryGenerator
Error in cling::AutoloadingVisitor::InsertIntoAutoloadingState:
   Missing FileEntry for TRestGeant4Track.h
   requested to autoload type TRestGeant4Track
Error in cling::AutoloadingVisitor::InsertIntoAutoloadingState:
   Missing FileEntry for TRestGeant4VetoAnalysisProcess.h
   requested to autoload type TRestGeant4VetoAnalysisProcess
Error in cling::AutoloadingVisitor::InsertIntoAutoloadingState:
   Missing FileEntry for TRestGeant4ToDetectorHitsProcess.h
   requested to autoload type TRestGeant4ToDetectorHitsProcess
Loading library : libRestGeant4.dylib
Loading library : libRestFramework.dylib
Loading library : libRestConnectors.dylib
Loading library : libRestDetector.dylib
In file included from input_line_153:1:
/Users/javi/git/rest/install/macros/REST_ViewEvents.C:4:10: fatal error: 'TRestDetectorSignalEvent.h' file not found
#include "TRestDetectorSignalEvent.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
CINT_TRestDetectorHitsEvent dictionary payload:5:10: fatal error: 'TRestDetectorHitsEvent.h' file not found
#include "TRestDetectorHitsEvent.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~
Error in <TInterpreter::AutoParse>: Error parsing payload code for class TRestDetectorHitsEvent with content:

#line 1 "CINT_TRestDetectorHitsEvent dictionary payload"


#define _BACKWARD_BACKWARD_WARNING_H
// Inline headers
#include "TRestDetectorHitsEvent.h"

#undef  _BACKWARD_BACKWARD_WARNING_H

In file included from input_line_175:1:
/Users/javi/git/rest/install/macros/REST_HitCentering.C:52:42: error: allocation of incomplete type 'TRestDetectorHitsEvent'
        TRestDetectorHitsEvent* ev = new TRestDetectorHitsEvent();
                                         ^~~~~~~~~~~~~~~~~~~~~~
CINT_TRestDetectorHitsEvent dictionary forward declarations' payload:5:77: note: forward declaration of 'TRestDetectorHitsEvent'
class __attribute__((annotate("$clingAutoload$TRestDetectorHitsEvent.h")))  TRestDetectorHitsEvent;
                                                                            ^
In file included from input_line_175:1:
/Users/javi/git/rest/install/macros/REST_HitCentering.C:54:28: error: cannot initialize a parameter of type 'TRestEvent *' with an lvalue of type 'TRestDetectorHitsEvent *'
        run->SetInputEvent(ev);
                           ^~
/Users/javi/git/rest/install/include/TRestRun.h:186:36: note: passing argument to parameter 'eve' here

Looks as some kind of linking problem ... needs investigation

REST standard pressure units

I am wondering if the standard units for pressure is atm or bar. They have slight difference. In TRestDetectorGas it stores pressure in atm, but in TRestSystemOfUnits the units with scale 1 is bar. Also in TRestSystemOfUnits the conversion to torr is wrong. 760 torr = 1 atm, not 1 bar

rename `any` to `RESTany`

It could be confusing to use the alias any for TRestReflector. Since there is std::any introduced in c++17. We may consider rename it to RESTany. In this case, all the processes needs to be updated

Problem using TRestRun to open a fresh generated file while processing data

This problem appeared at rest-for-physics/axionlib#12

I suspect the problem is related to the intermediate encapsulation added by TRestAxionEventProcess::TRestEventProcess added so that we can add a common feature to any axion event process, this approach might be handy at other libraries.

When I open the file using TFile directly, I am able to access the information registered. No problem there.

The problem appears when using TRestRun *r = new TRestRun("file.root") or restRoot file.root

 *** Break *** segmentation violation



===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0  0x00007fc769f53437 in __GI___waitpid (pid=41451, stat_loc=stat_loc
entry=0x7ffdb9dcc6d8, options=options
entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1  0x00007fc769ed15df in do_system (line=<optimized out>) at ../sysdeps/posix/system.c:149
#2  0x00007fc76c605cb3 in TUnixSystem::StackTrace() () from /programas/root/6.25.01-minuit2/lib/libCore.so
#3  0x00007fc76c6085b5 in TUnixSystem::DispatchSignals(ESignals) () from /programas/root/6.25.01-minuit2/lib/libCore.so
#4  <signal handler called>
#5  0x00007fc76afdc655 in ROOT::delete_TRandom3(void*) () from /programas/root/6.25.01-minuit2/lib/libMathCore.so
#6  0x00007fc76c00923b in TBufferFile::ReadFastArray(void**, TClass const*, int, bool, TMemberStreamer*, TClass const*) () from /programas/root/6.25.01-minuit2/lib/libRIO.so
#7  0x00007fc76c25171f in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /programas/root/6.25.01-minuit2/lib/libRIO.so
#8  0x00007fc76c0d3ccd in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /programas/root/6.25.01-minuit2/lib/libRIO.so
#9  0x00007fc76c00feca in TBufferFile::ReadClassBuffer(TClass const*, void*, TClass const*) () from /programas/root/6.25.01-minuit2/lib/libRIO.so
#10 0x00007fc76c0ab944 in TKey::ReadObj() () from /programas/root/6.25.01-minuit2/lib/libRIO.so
#11 0x00007fc76c05c170 in TDirectoryFile::Get(char const*) () from /programas/root/6.25.01-minuit2/lib/libRIO.so
#12 0x00007fc76c9ce8ff in TRestRun::ReadInputFileMetadata() () from /home/jgalan/rest-framework/install/lib/libRestFramework.so
#13 0x00007fc76c9d7b5a in TRestRun::OpenInputFile(TString, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) () from /home/jgalan/rest-framework/install/lib/libRestFramework.so
#14 0x00007fc76c9d7f2a in TRestRun::TRestRun(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) () from /home/jgalan/rest-framework/install/lib/libRestFramework.so
#15 0x000055952d61a25a in main ()
===========================================================


The lines below might hint at the cause of the crash.
You may get help by asking at the ROOT forum https://root.cern.ch/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at https://root.cern.ch/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#5  0x00007fc76afdc655 in ROOT::delete_TRandom3(void*) () from /programas/root/6.25.01-minuit2/lib/libMathCore.so
#6  0x00007fc76c00923b in TBufferFile::ReadFastArray(void**, TClass const*, int, bool, TMemberStreamer*, TClass const*) () from /programas/root/6.25.01-minuit2/lib/libRIO.so
#7  0x00007fc76c25171f in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /programas/root/6.25.01-minuit2/lib/libRIO.so
#8  0x00007fc76c0d3ccd in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /programas/root/6.25.01-minuit2/lib/libRIO.so
#9  0x00007fc76c00feca in TBufferFile::ReadClassBuffer(TClass const*, void*, TClass const*) () from /programas/root/6.25.01-minuit2/lib/libRIO.so
#10 0x00007fc76c0ab944 in TKey::ReadObj() () from /programas/root/6.25.01-minuit2/lib/libRIO.so
#11 0x00007fc76c05c170 in TDirectoryFile::Get(char const*) () from /programas/root/6.25.01-minuit2/lib/libRIO.so
#12 0x00007fc76c9ce8ff in TRestRun::ReadInputFileMetadata() () from /home/jgalan/rest-framework/install/lib/libRestFramework.so
#13 0x00007fc76c9d7b5a in TRestRun::OpenInputFile(TString, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) () from /home/jgalan/rest-framework/install/lib/libRestFramework.so
#14 0x00007fc76c9d7f2a in TRestRun::TRestRun(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) () from /home/jgalan/rest-framework/install/lib/libRestFramework.so
#15 0x000055952d61a25a in main ()
===========================================================

I will keep investigating but any help is welcome. @nkx @rest-for-physics/core_dev

Move project folder structure to a more comprehensive way

While I was creating AlphaCAMM project I found that the projects in the repository https://github.com/rest-for-physics/framework/tree/master/projects are split in different subfolders (projects) that doesn't follow a comprehensive structure, e.g. for IAXO we have:
iaxo-analysis
iaxo-simulations
iaxo-geometry
iaxo-readouts

I would suggest to create a folder (project) named iaxo and create inside the different sub-folders: analysis, simulations, geometry and readouts. I am working in a project for alphaCAMM that we can use as template or at least we can iterate over it.

TRestFitAnalysisProcess needs to be implemented

We are missing a generic process that just takes in the configuration file an observable from the analysis tree to be the target of the fit and define a fit function, range and parameter hints and limits.

The process should store the results of the fit.

This process could be handy when for example we study the gain evolution of a particular dataset from the detector real data. We could for instance use this process to get the position of a particular calibration peak found at a given observable.

We could have something as ...

<TRestFitAnalysisProcess>

    <fit observable="hitsAna_energy" func="gaus" range="(100,400)" initParam=.... limitParam=..../>
    <fit observable="rawAna_thresholdIntegral" func="[0] + [1]*x + [2]*TMath::Exp(..." />

</TRestFitAnalysisProcess>

clean-state.sh improve output. Make it more user friendly.

When we execute this script, if there are submodules where we have no access, it will output errors when trying to pull from those non-authorised repositories.

Even if we have access to all submodules this script will produce a lot of info. This script could be more sophisticated and control the output that throws on screen by using something similar to pull-submodules.py script.

TRestRun initialization from RML not working properly

#129 introduces a testing framework. On that PR I have also introduced a simple test which should load a basic RML and check if the serialization works properly.

I am using TRestRun for this since I think its the simplest class that can perform this job.

However it doesn't seem to work and we don't know why.

Usage:

TRestRun run("test.rml"); run.PrintMetadata();

The printed metadata will correspond to an unitialized run, it won't load values form file.

root [0] TRestRun run("test.rml")
(TRestRun &) Name:  Title:
root [1] run.PrintMetadata()
                                                        ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
                                                        ||                                        TRestRun content                                        ||
                                                        ||                                       Config file : null                                       ||
                                                        ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
                                                        ||                                            Name :                                              ||
                                                        ||                                            Title :                                             ||
                                                        ||                                     REST Version : 2.3.10                                      ||
                                                        ||                                   REST Official release: No                                    ||
                                                        ||                                        Clean state: No                                         ||
                                                        ||                                     REST Commit : 252286d2                                     ||
                                                        ||                                    REST Library version : 0                                    ||
                                                        ----------------------------------------------------------------------------------------------------
                                                        ||                                        Version : 2.3.10                                        ||
                                                        ||                                     Parent run number : 0                                      ||
                                                        ||                                         Run number : 0                                         ||
                                                        ||                                   Experiment/project : Null                                    ||
                                                        ||                                        Run type : Null                                         ||
                                                        ||                                         Run tag : Null                                         ||
                                                        ||                                        Run user : root                                         ||
                                                        ||                                     Run description : Null                                     ||
                                                        ||                       Start Date/Time : 2022-3-23 11:35:41 (1.64804e+09)                       ||
                                                        ||                        End Date/Time : 2022-3-23 11:35:27 (1.64804e+09)                        ||
                                                        ||                                       Input file : null                                        ||
                                                        ||                                Output file : rest_default.root                                 ||
                                                        ||                                     Number of events : -1                                      ||
                                                        ----------------------------------------------------------------------------------------------------

test.rml contents:

<?xml version="4.0" encoding="UTF-8" standalone="no" ?>
<test>
    <globals>
        <parameter name="globalVariable" value="global"/>
    </globals>
    <TRestRun name="Basic Run" title="Basic Run Title">
        <parameter name="experimentName" value="TRestRun Basic Test"/>
        <parameter name="runType" value="Test"/>
        <parameter name="runNumber" value="1"/>
        <parameter name="runTag" value="Test"/>
        <parameter name="outputFile" value="Run[fRunNumber]_[fRunTag]_[fExperimentName].root"/>
        <parameter name="runDescription" value="This is a test for TRestRun"/>
        <parameter name="verboseLevel" value="3"/>
        <parameter name="overwrite" value="off"/>
        <parameter name="readOnly" value="false"/>
    </TRestRun>
</test>

Could @nkx111 please try to fix this? I think you're the best person to do this. Thanks!

cmake system sometimes does not add geant4.sh to thisREST.sh

It has been observed that sometimes the cmake system identifies a Geant4 installation, but it does not add properly the geant4.sh script to be loaded at thisREST.sh.

This is probably linked with compiling REST without loading Geant4 using previously source /../../geant4.sh but cmake system properly identifying a Geant4 installation and going all the way through without imposing the geant4 version at thisREST.sh script.

It would be good to fix this problem to avoid problems connected with the following forum topic:

http://rest-forum.unizar.es/t/restg4-undefined-symbol-error-in-v2-3-8-official-release/499/2

Modify thisREST.cmake to restore previous alias macro functionality

Previous to version 2.3 all macros were stored inside the /macros directory and were transformed into aliases via

foreach(mac ${rest_macros})

string(REPLACE " " "" mac ${mac})
string(REPLACE "rest" "" m ${mac})

in the thisREST.cmake file.

However since in the new version the modules such as restG4 are split into submodules (different git respositories), this macro cannot obtain the name of the macros in order to put them into aliases (such as restViewG4Event now should be restManager REST_Geant4_ViewEvent).

One solution would be to hardcode the aliases into the thisREST.cmake file but perhaps there is a more elegant solution that won't break once the macro names change again.

Extending InitFromConfigFile scheme used in REST processes to any metadata class

I wonder if we could extend the nice system we have inside TRestEventProcess to TRestMetadata (i.e. avoiding re-implementation of InitFromConfigFile for simple parameters).

I thought it could be as simple as implementing TRestMetadata method,

    virtual void InitFromConfigFile() {
        map<string, string> parameters = GetParametersList();

        for (auto& p : parameters) p.second = ReplaceMathematicalExpressions(p.second);

        ReadParametersList(parameters);
    }

however, I run into troubles.

Any quick solution to extend the processes InitFromConfigFile strategy to any metadata class?

Improve cmake libraries compilation system

Following the forum discussion with @nkx111 at

http://rest-forum.unizar.es/t/put-cmake-commands-to-each-library-cmakelists-in-v2-3/433

It would be interesting to improve the cmake library adding system.

The main CMakeLists.txt at sources/ could just call to add_subdirectory(library/name). Then, any other definitions could be placed at the library CMakeLists.txt itself.

It could be even improved to loop on all subdirectories found at source/libraries and if RESTLIB_NAME was enabled, then add the subdirectory libraries/name.

Some observables are not multithread safe

The observable sAna_MeanRate_InHz gives different results when the data analysis is performed in multithread mode than when it is done using a single thread. It's true that we found some event ID issues when multithreading and we introduced some warnings that come out during the analysis. So maybe it is related to this.
image

Not sure if it is possible to force a process to be single-threaded. That observable should probably be defined at a dedicated process, e.g. TRestRateAnalysisProcess. There might be other observables affected, as timeDelay.

TRestMapGroup and TRestMapGroupProcess implementation

It would be good to design and implement a generic process that would allow to perform calculations on a selection of elements inside a std::map.

The most clear example right now is when I have a system of vetos covering the detector. Those vetos are registered as signals, analysed using TRestRawSignalAnalysisProcess and few observables written together into std::map variables at the analysis tree. Thats the validation pipeline introduced at rest-for-physics/rawlib#25.

Now, I want to be able to define groups of signals, so that I would be able to define a subset of signal ids over which I will perform some calculations. So TRestMapGroup would allow me to give a name to the group and define the elements from std::map.first that belong to that group. I would be able to define any number of groups. This is somehow done now at TRestRawVetoAnalysisProcess.

TRestMapGroupProcess would allow me to use a TRestMapGroup definition to perform calculations on the specific subsets. The user would define at the process level which operations wants to be performed on the subset, i.e. average, max value, deviation, etc.

Those calculated values would be written to the process as pcsName_groupName_average, pcsName_groupName_deviation, ...

TRestAnalysisPlot problems using `classify`

Hi, I am trying to produce a plot using the following RML definition:

  <TRestAnalysisPlot name="restplot" title="Basic Plots">

    <parameter name="previewPlot" value="True"/>

    <canvas size="(1000,800)" divide="(1,1)" save="plots.root"/>

    <plot name="Energy" title="Energy spectrum" xlabel="Energy [keV]" ylabel="Counts" value="ON" stats="ON" >

      <histo name="1MeV_5um" fillColor="gray" lineColor="red">
        <variable name="g4Ana_totalEdep" range="(0,10000)" nbins="100"/>
        <classify runTag="1MeV_5um"/>
      </histo>

      <histo name="5MeV_5um" fillColor="gray" lineColor="blue">
        <variable name="g4Ana_totalEdep" range="(0,10000)" nbins="100"/>
        <classify runTag="5MeV_5um"/>
      </histo>

      <histo name="5MeV_1um" fillColor="gray" lineColor="black">
        <variable name="g4Ana_totalEdep" range="(0,10000)" nbins="100"/>
        <classify runTag="5MeV_1um"/>
      </histo>

    </plot>

If I introduce a bit of debugging inside TRestAnalysisPlot

            bool firstdraw = false;
            TH3F* hTotal = hist.ptr;
            for (unsigned int j = 0; j < fRunInputFileName.size(); j++) {
                cout << "File " << fRunInputFileName[j] << endl;
                auto run = GetRunInfo(fRunInputFileName[j]);
                // apply "classify" condition
                bool flag = true;
                auto iter = hist.classifyMap.begin();
                while (iter != hist.classifyMap.end()) {
                    cout << "first : " << iter->first << endl;
                    cout << "second : " << iter->second << endl;
                    cout << "third : " << run->GetRunInformation(iter->first) << endl;
+-------  4 lines: if (run->GetRunInformation(iter->first) != iter->second) {------------------------------------------------------------------------------------------------
                    iter++;
                }
                cout << "Continue?" << endl;

Then, it seems the file is a new one, but the runTag value is not updating.

Screenshot 2022-01-13 at 12 31 44

If I open the files using restRoot and then inspect run0->GetRunTag I get the right value.

Probably a bug?

Data structures are difficult to handle

Current code design makes difficult to handle data structures, e.g. in TRestHits.h

class TRestHits : public TObject {
   public:
    Int_t fNHits;         ///< Number of punctual energy depositions, it is the length
                          ///< for all the array
    Double_t fTotEnergy;  ///< Event total energy

    std::vector<Float_t> fX;          // [fNHits] Position on X axis for each punctual
                                      // deposition (units mm)
    std::vector<Float_t> fY;          // [fNHits] Position on Y axis for each punctual
                                      // deposition (units mm)
    std::vector<Float_t> fZ;          // [fNHits] Position on Z axis for each punctual
                                      // deposition (units mm)
    std::vector<Float_t> fT;          // [fNHits] Absolute time information for each punctual deposition
                                      // (units us, 0 is time of decay)
    std::vector<Float_t> fEnergy;     // [fNHits] Energy deposited at each
                                      // 3-coordinate position (units keV)
    std::vector<REST_HitType> fType;  //

Y would suggest to move to a different data structure e.g.

  class TRestHit : public TObject {
    Float_t fX;
    Float_t fY;
    Float_t fZ;
    Float_t fT;
    Float_t fEnergy;
    REST_HitType fType;
   enum REST_HitType { unknown = -1, X = 2, Y = 3, Z = 5, XY = 6, XZ = 10, YZ = 15, XYZ = 30 };
  };

 class TRestHits : public TObject {
   public:
     //fNHits is not needed anymore, it is just hitArray.size( );
     Double_t fTotEnergy;  ///< Event total energy
     std::vector<TRestHit> hitArray;          // Array of hits
 

This would simplify the handling of TRestHits using standard std iterators, for instance this part of the code mimic std iterators:

  class TRestHits_Iterator : public std::iterator<std::random_access_iterator_tag, TRestHits_Iterator> {

The implementatio of this proposal would imply a lot of changes in the code and perhaps the root files might loose the backward compatibility, but it will be a big improvement that would simplify the code.

Let me know what do you think, I can help with the migration.

Create a better unified drawing methods

There are some features that could be shared between the DrawEvent methods.

For example, option arguments retrieval could be common. We could implement a common Drawing for event types using TRestHits.

Perhaps, it could be implemented at TRestEvent::DrawHitsEvent level the retrieval of options. Then, at TRestDetectorHitsEvent and TRestGeant4Event we could do:

TRestDetectorHitsEvent::DrawEvent( opt )
{
         TRestEvent::DrawHitsEvent( opt );

         // Write now specific drawing 
}

Another option is to create a new class TRestEventDrawer, that contains dedicated methods and the common drawing routines. Then TRestEvent could inherit from TRestEvent::TRestEventDrawer

This was discussed at rest-for-physics/detectorlib#33

Add the possibility to load RML files using restRoot

It would be interesting to automatically load the class definitions found inside a RML file directly using restRoot.

It could just create an instance md_name for each object found at the RML.

So that we could do restRoot config.rml

Introduce a testing framework

Do the necessary steps to introduce a testing framework to better test our code (it currently only runs integration tests).

The proposed testing framework of choice is Google Test.

Review of restDecay0

It seems to me that the class found at packages/restDecay0 would be better part of the geant4 library.

This would require a renaming: TRestParticleCollectionDecay0 --> TRestGeant4ParticleCollectionDecay0.

Then, the geant4 library should have just a compilation option to enable this. Just as we enable other features as REST_SQL or REST_EVE. We could have another REST_DECAY0 that enables this feature and makes RestGeant4 to link against Decay0. This kind of options I would refer it as a plugin, it is a class that extends the framework or library functionality.

A packages is more a standalone program, with its own executable, that links and makes use of REST.

Implementation of a generic TRestEventSelectionProcess

It might happen that we process a ROOT file once using a very quick/preliminary analysis. Then, we want to apply a more sophisticated analysis to events that satisfy certain conditions in the resulting file at the analysisTree, even at a late processing stage. In the second more sophisticated and computationally expensive processing chain we may add processes that require more computation. In other words, we will re-process with more detail but for a fewer amount of events.

Thus we need a process that accepts in its definition a list of event ids, or a range of event ids. The case could be extended to other basic event identifiers, as sub-event id, or event Tag, timestamp, ... The process in each event loop will compare the event ID against the ones that are in the list. If the ID is not found inside the list it will just return return NULL.

This process could be as simple as just retrieving the event list from a text file, a ROOT TTree, or directly at the RML.

Or, it could become more complex and receive as input the ROOT file containing the analysisTree, and define the selection rules that will generate the event ID list inside the InitProcess.

Development branch pipeline is failing due to detector library update.

Hi, I found that the upgrade of the detector library to v1.1 at the development branch causes wrong results at pandax-iii MC pipelines.

It is a bit tricky because if the problem is found, in order to solve the issue at the pipeline, the detector library must be increased to version 1.2 and added to the main framework. Since the framework pipeline is running with the "official" library linked.

Plz @nkx could you have a look to check why the pandax-iii MC Pipeline is failing?

Implement automatic parameter loading for all the metadata classes

It is time to implement automatic parameter loading from rml text to all the class data members. We shall verify each metadata classes that:

  1. All the data members are named following our naming convention
  2. The method InitFromConfigFile() has only simple logic of parameter reading: fXXX=StringToDouble(GetParameter("xXX","10"))
  3. The data members have default values

TRestAnalysisPlot simplification using RDataFrame

A lot of efforts have been put so that TRestAnalysisPlot works efficiently opening thousands of files at once without major leak problems, and efficiently. So, right now, TRestAnalysisPlot is quite robust and efficient.

However, rewriting TRestAnalysisPlot using RDataFrame could strongly simplify/reduce the code.

Serialization/Deserialization of RML files from/to YAML

I think YAML is an overall better format for our use case compared to XML, the only read advantage of XML being that we are used to it.

We can make use of a schema to help the user correctly write / modify an RML file. (this can also be done with XML if anyone dares...).

a PoC is available at https://github.com/lobis/radiation-transport/tree/main/libraries/Simulation/Config, as you can see the amount of code needed is much smaller than what we need to TRestMetadata.

There are two ways to do this:

  • Extend (rewrite) our TRestMetadata classes to support both XML and YAML files. I think this would add too much complexity to our already complex code.

  • Write a parallel TRestMetadata (TRestConfiguration ?) in an experimental namespace and work to support all current configurations, translate all RML files in the framework and once it is stablished deprecate the XML files.

I like the second option the most but it is up for discussion.

image

TRestDataSet and TRestDiscrimination implementation

We need to define new classes that allow us to create datasets. A dataset is a class that allows to define some rules and a period of time to filter a set of ROOT files in a path containing all the relevant data. A dataset will contain rules to define background files, and rules to define calibration files. Obviously those datasets could have been generated using Monte Carlo or real data. A dataset, therefore, will identify which calibration data is used for pattern recognition for each background population.

TRestDataSet will use RDataFrame to quickly define a glob pattern that identifies the files, and require that they fulfil certain metadata conditions at any of the classes stored inside the ROOT files. This will create a dataset, i.e. a combined TTree (or RDataFrame) with all the matching files.

This might need a bit of additional brainstorming, however, TRestDataSet should at least:

  • At initialisation it should define the metadata rules for accepting a ROOT file or not (i.e. a subrun).
  • It should keep track of the total time (duration) of all the files added.
  • Define a method to set the variables that will be found at final tree (that we will be able to export later on). I.e: Int_t SetObservables( std::string ).
  • Implement a method to define which variables will be used as discriminant. I.e: Int_t SetDiscriminants( std::string ).
  • A method to define a range in each observable. SetDiscriminantLimits( Double_t efficiency);. This method will define a big multi-dimensional box and it will start to reduce its limits in order to find the optimum box that keeps the fraction of the calibration events given by efficiency inside.

Another class named TRestDiscrimination will define an interface with pure abstract classes to define the methods that MUST be implemented in the inherited class. The TRestDiscrimination will contain a vector of TRestDataSets.

TRestDiscrimination will allow us to add any number of datasets. And call some methods to retrieve the final background population selected.

This is still to be reflected a bit, however, TRestDiscrimination should at least implement:

  • A pure virtual method to get the resulting population after applying the cuts at the particular method: RDataRFrame GetSelection( );.
  • A pure virtual method to get the optimum efficiency that maximises the signal-to-background relation: GetOptimumEfficiency

A first specific discrimination class will be TRestBasicDiscrimination. This class will perform an event selection following a basic rectangular cuts scheme with an optimum for the dataset.

There is a working document at the following location to be used as a draft:

https://docs.google.com/presentation/d/1gQPMpR-wcQgLzKMNVgdXvgVSXPyQOhbirpx_vcSL1vo/edit?usp=sharing

As soon as this issue is open do not hesitate to participate or contribute to this topic! New contributors are welcome to join discussions or development.

See also issue #346

Remove using namespace std from header files

Several instances to using namespace std are found in several header files, which is considered bad coding practise:
https://stackoverflow.com/questions/1452721/why-is-using-namespace-std-considered-bad-practice

Here a list of the header files where I have found using namespace std in the code:

./source/framework/core/inc/TRestTask.h
./source/framework/tools/inc/TRestStringHelper.h
./source/framework/tools/inc/TRestSystemOfUnits.h
./source/framework/tools/inc/TRestDataBase.h
./source/framework/tools/inc/TRestPhysics.h
./source/framework/tools/inc/TRestReflector.h
./source/framework/tools/inc/TRestStringOutput.h
./source/framework/tools/inc/TRestTools.h
./source/packages/restG4/include/PhysicsList.h
./source/packages/restG4/include/EventAction.h
./source/packages/restG4/include/SteppingAction.h
./source/packages/restG4/include/RunAction.h
./source/packages/restG4/include/PrimaryGeneratorAction.h

However the required changes in the code could be larger than these header files, as has been pointed in #124 for TRestAnalysisPlot.h

I am willing to give a hand on this issue, but I think we should divide the work because the load can be large.

Updating TRestRawVetoAnalysisProcess and adding TRestRawVetoCalibrationProcess

I thought about following changes to the veto analysis and adding a process for the veto calibration:

  1. changing the way how veto groups are handled by TRestRawVetoAnalysisProcess: keep only two observables veto_PeakTime and veto_MaxPeakAmplitude, where each is a std::map with all veto signal IDs as keys, rather than two observables for each group. This makes further processing simpler.
  2. Adding an optional parameter, that contains a std::map which defines the veto groups, e.g <1234, "top_1"> and can be read out when necessary for the analysis / plotting
  3. Adding a process TRestRawVetoCalibrationProcess which adds an observable with the calibrated veto amplitudes. The calibration factors can be defined in the rml, or can be loaded from some public file. In this process also the veto groups can be defined.
  4. The calibration itself (to obtain the calibration factors) is done offline, but I can share the macro later in the iaxo-analysis git.

Does this sound good to you, or has anyone suggestions to do it differently?

Increasing minimum CMake version causes problems on install

When working on #129 I noticed that the pipeline was not passing due to the installation directory missing some files (the binaries and libraries). After much trial and error, I found that the problem was that I had increased the minimum required version of CMake. After reverting this change, it was fixed 848f2da.

However since #129 is using some new features of CMake (and perhaps other code is already doing the same), having 3.5 as the minimum CMake version (which is from 6 years ago), will probably lead to problems on systems that use a cmake version closer to 3.5.

I vote to bump the minimum CMake version to 3.16, but in order to do that we need to fix this issue.

TRestAnalysisPlot feature allowing to plot accumulative map observables inside a TRestAnalysisTree

This feature would allow to create for example a channel activity weighted with different detector parameters.

I.e. if we have the risetime value versus the channel ID at our analysisTree, then it would be interesting to show the average rise time per channel in a single plot, either average or accumulated total.

We could produce an activity channel weighted with the energy using the map variable holding the pulse amplitude versus channel ID.

And so on. We are not able yet to do this kind of plots directly with TRestAnalysisPlot.

CMake add an option to enable all libraries

We need to upgrade the cmake library system so that we are able to enable all libraries but still remove one library from the list.

For example, lets imagine I want to add all libraries for compilation except for the geant4 library. Then, I would be able to do:

cmake --all-libs -DRESTLIB_GEANT4=OFF ../

or

cmake -DREST_ALL_LIBS=ON -DRESTLIB_GEANT4=OFF ../

When using such a cmake construction, the REST_ALL_LIBS=ON or --all-libs definitions should enable all the list of variables that start by RESTLIB_.

I found a solution to this problem here:
https://gitlab.kitware.com/cmake/community/-/wikis/doc/tutorials/SettingVariableGroups

However, I was not lucky on a first try.

There is a problem with REST_ALL_LIBS being cached. And even if I do:

cmake -DREST_ALL_LIBS=ON ../

and then

cmake -DRESTLIB_GEANT4=OFF ../

The second cmake execution will have no effect, since REST_ALL_LIBS will still be ON.

TRestReflector::ToString() does not correctly convert double

https://github.com/rest-for-physics/framework/blob/master/source/framework/tools/src/TRestReflector.cxx

The current implementation:

string TRestReflector::ToString() {
    if (type == "string") return *(string*)(address);
    if (address == nullptr) return "null";
    RESTVirtualConverter* converter = RESTConverterMethodBase[type];
    if (converter != nullptr) {
        return converter->ToString(address);
    } else {
        return Form("Type: %s, Address: %p", type.c_str(), address);
    }
}

There is a problem when converting doubles such as timestamp. The output is truncated in such a way that the full number cannot be recovered. This is becaues the cout also doesn't display the full number, the cout.precesion needs to be bigger to correctly display this numbers.

Pipe validation to check initialisation code in metadata classes

All metadata classes must define at its initialisation stage the corresponding class name, and the LIBRARY_VERSION. Which is usually as follows:

void TRestMetadataClass::Initialize() {
    SetSectionName(this->ClassName());
    SetLibraryVersion(LIBRARY_VERSION);
}

We need a script running at the pipeline before the build stage to assure this is as that. This will only be tested at metadata classes under source/libraries. It could be inspired by pipeline/validateProcesses.py

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.