Branch	Travis CI	Coveralls	Codecov	Codacy	Landscape	CodeClimate
master
develop

• gem-plotting-tools
  • Setup
    • Setup at Point 5
  • Masking Channels Algorithmically
    • Definitions
    • Deriving Channel Configuration
    • Providing Cuts for maskReason at Runtime
  • List Of Scandate Input Files
    • Two Column Format
    • Three Column Format
    • Automatically Generating Set of listOfScanDates.txt
  • Analyzing Scans:
    • Analyzing Python Ultra Scan Data
      • plot_eff.py
      • plot_eff.py Arguments
      • plot_eff.py Example
      • plot_eff.py Input File
    • Analyzing xDAQ Scan Data
  • Arbitray Plotting Tools
    • gemPlotter.py
      • gemPlotter.py Arguments
      • gemPlotter.py Input File
      • gemPlotter.py Example: Making a time series with plotTimeSeries.py
      • gemPlotter.py Example: Making a 1D Plot - Channel Level
      • gemPlotter.py Example: Making a 1D Plot - VFAT Level
      • gemPlotter.py Example: Making a 2D Plot
    • gemTreeDrawWrapper.py
      • gemTreeDrawWrapper.py Arguments
      • gemTreeDrawWrapper.py Input File
      • gemTreeDrawWrapper.py Example: Making a Plot
      • gemTreeDrawWrapper.py Example: Fitting a Plot
  • Scurve Plotting Tools
    • gemSCurveAnaToolkit.py
      • gemSCurveAnaToolkit.py Arguments
      • gemSCurveAnaToolkit.py Input File
      • gemSCurveAnaToolkit.py Example: Making a Plot
    • Comparing Scurves Results Across Scandates: plotSCurveFitResults.py
      • plotSCurveFitResults.py Arguments
      • plotSCurveFitResults.py Input File
      • plotSCurveFitResults.py Example
    • Analyzing the Time Evolution of Channels: timeHistoryAnalyzer.py
      • timeHistoryAnalyzer.py Arguments
      • timeHistoryAnalyzer.py Examples
  • Packaging Tool: packageFiles4Docker.py
    • packageFiles4Docker.py Arguments
    • packageFiles4Docker.py Input Files
    • packageFiles4Docker.py Example
  • Cluster Computing Tools
    • Cluster Analysis of S-Curve Data: clusterAnaScurve.py
      • clusterAnaScurve.py: Arguments
      • Full Example For P5 S-Curve Data

Created by gh-md-toc

The $SHELL variable $ELOG_PATH should be defined:

export ELOG_PATH=/your/favorite/elog/path

Also a useful $SHELL variable is $BUILD_HOME which should be the directory at the start of your working directory. Checkout the sw_utils repository by executing:

cd $BUILD_HOME
git clone https://github.com/cms-gem-daq-project/sw_utils.git

Then execute:

source sw_utils/scripts/setup_gemdaq.sh -c <cmsgemos tag> -g <gem-plotting tag> -G <gem-plotting dev version optional>

Tags for each of the repo's can be found:

• cmsgemos version X.Y.Z (-c X.Y.Z)
• gemplotting version X.Y.Z-devA (-g X.Y.Z -G A)

Where X, Y, Z, and A are integers, and most likely will be different for each of the repositories. If a development version is not to be used (normal case), you can drop the -G option. If this is the first time you are executing the above command, it will create a Python virtualenv for you and install the cmsgemos and gemplotting packages. It may take some time to download them, so be patient and do not interrupt the installation.

Example

source setup_gemdaq.sh -c 0.3.1 -g 1.0.0 -G 5

This command will install the following packages:

• cmsgemos version 0.3.1 (-c 0.3.1)
• gemplotting version 1.0.0-dev5 (-g 1.0.0 -G 5)

In addition to installing the dependencies, the script will try to guess $DATA_PATH based on the machine you are using.

To disable the python env execute:

deactivate

To re-enable the python env, source the script again:

cd $BUILD_HOME
source sw_utils/scripts/setup_gemdaq.sh

Note that you should always source the setup script from the same directory.

At P5, gem-plotting-tools is installed system-wide. Setting it up is as simple as:

source /nfshome0/gempro/bin/get_gem_env.sh

This command should be run every time you connect. You can put it in your .bashrc or .bash_profile so it's done automatically.

When the analysis software decides a channel should be masked it is because it falls under one of the categories defined in the MaskReason class of anaInfo.py. Multiple reasons can be assigned to a channel for why it is masked, and the total maskReason is a 5-bit binary number. Presently these reasons are:

The scurve sigma is the sigma of the modified error function used to fit the s-curve measurements. It comes from the TF1 object used to fit scurves in ScanDataFitter::fit() of fitScanData.py.

A channel's effective pedestal is the percent of time a channel's comparator fires when injected charge is zero. This is determined from an s-curve measurement via:

effPed = scurve_fit_func.Eval(0) / n_pulses

Where n_pulses are the number of charge injections for a given DAC value performed by the calibration module.

The analysis software will record the maskReason in decimal reprementation. So for example a channel having maskReason = 24 corresponds to 0b11000 which means the channel was assigned the HighEffPed and HighNoise maskReasons.

The following procedure is used, note these steps must be executed one after another, without LV power cycle or action to cause a reset of the VFAT settings (e.g. SCA reset):

Please note that while DeadChannel is given in maskReason these channels are never masked such that they can be tracked overtime.

If a channel was masked at the time of acquisition of a test involving an s-curve measurement (e.g. trimChamber(V3).py or ultraScurve.py) then it will be assigned the FitFailed reason since the original reason is not known without referencing a previous scan.

When analyzing the above s-curves taken by trimChamber(V3).py The following command line arguments are available for specifying the cut values for assigning the DeadChannel, HighNoise, and HighEffPed pedestal.

Many of the tools found in the macros/ directory require a listOfScanDates.txt file. These come in either two or three column versions and the parseListOfScanDatesFile(...) of anautilities.py is designed to parse either version and provide the tool with the correct information. This means that, baring other command line arguments, the two formats are relatively interchangeable.

This should be a tab deliminited text file. The first line of this file should be a list of column headers formatted as:

ChamberName scandate

Subsequent lines of this file are the values that correspond to these column headings. The value of the ChamberName column must correspond to the value of one entry in the chamber_config dictionary found in mapping/chamberInfo.py. The next column is for scandate values. Please note the # character is understood as a comment, lines starting with a # will be skipped.

A complete example for a single detector is given as:

ChamberName scandate
GE11-VI-L-CERN-0001    2017.08.11.16.30
GE11-VI-L-CERN-0001    2017.08.14.20.54
GE11-VI-L-CERN-0001    2017.08.30.15.03
GE11-VI-L-CERN-0001    2017.08.30.21.39
GE11-VI-L-CERN-0001    2017.08.31.08.28
GE11-VI-L-CERN-0001    2017.08.31.15.46
GE11-VI-L-CERN-0001    2017.09.05.11.41
GE11-VI-L-CERN-0001    2017.09.12.14.24
GE11-VI-L-CERN-0001    2017.09.13.16.45

This should be a tab deliminited text file. The first line of this file should be a list of column headers formatted as:

ChamberName scandate    <Indep. Variable Name>

Subsequent lines of this file are the values that correspond to these column headings. The value of the ChamberName column must correspond to the value of one entry in the chamber_config dictionary found in mapping/chamberInfo.py. The Indep. Variable Name is the independent variable that --branchName will be plotted against, if it is not numeric please use the --alphaLabels command line option. Please note the # character is understood as a comment, lines starting with a # will be skipped.

A complete example for a single detector is given as:

ChamberName scandate    VT_{1}
GE11-VI-L-CERN-0002 2017.09.04.20.12    10
GE11-VI-L-CERN-0002 2017.09.04.22.52    20
GE11-VI-L-CERN-0002 2017.09.05.01.33    30
GE11-VI-L-CERN-0002 2017.09.05.04.21    40
GE11-VI-L-CERN-0002 2017.09.05.07.11    50

Here the ChamberName is always GE11-VI-L-CERN-0002 and --branchName will be plotted against VT_{1} which is the Indep. Variable Name. Note the axis of interest will be assigned the label, with subscripts in this case, of VT_{1}.

A complete example for multiple detectors is given as:

ChamberName scandate    Layer
GEMINIm27L1 2019.09.04.20.12    GEMINIm27L1
GEMINIm27L2 2019.09.04.22.52    GEMINIm27L2
GEMINIm28L1 2019.09.05.01.33    GEMINIm28L1
GEMINIm28L2 2019.09.05.04.21    GEMINIm28L2
GEMINIp02L1 2019.09.05.07.11    GEMINIp02L1
GEMINIp02L2 2019.09.05.07.11    GEMINIp02L2

Here the ChamberName is different for each line and --branchName will be plotted against Layer. Note since the Indep. Variable Name is not numeric the command line option --alphaLabels must be used.

To automatically generate a set of listOfScanDates.txt files for all s-curve measurements for each of the chambers defined in chamber_config.values() of chamberInfo.py execute:

plotTimeSeries.py --listOfScanDatesOnly --startDate=2017.01.01

For each detector defined in chamber_config.values() the listOfScanDAtes.txt file will be found at:

$DATA_PATH/<ChamberName>/scurve/

If you are interested in generating a set of listOfScanDates.txt files for measurements other than scurves supply the --anaType argument at the time of execution like:

plotTimeSeries.py --listOfScanDatesOnly --startDate=2017.01.01 --anaType=<type>

The list of supported anaType's are from ana_config.keys() of anaInfo.py. In this case the listOfScanDAtes.txt file for each chamber will be found at:

$DATA_PATH/<ChamberName>/<anaType>/

Analysis is broken down into either analyzing data taken with the python ultra scan tools or with xdaq.

The following tools exist to help you to analyze scans taken with the ultra tools in the vfatqc-python-scripts repository:

• ana_scans.py,
• anaUltraLatency.py,
• anaUltraScurve.py, and
• anaUltraThreshold.py.

See extensive documentation written on the GEM DOC Twiki Page.

For some test stands where you have configured the input L1A to pass only through a specific point of a detector you can use the data taken by ultraLatency.py to calculate the efficiency of the detector. To help you perform this analysis the plot_eff.py tool has been created.

The following table shows the mandatory inputs that must be supplied to execute the script:

The following table shows the optional inputs that can be supplied when executing the script:

Note if the --bkgSub option is used then you must first call anaUltraLatency.py for each of the scandates given in the --infilename.

The format of this input file should follow the Three Column Format.

To calculate the efficiency using VFATs 12 & 13 in latency bin 39 for a list of scandates defined in listOfScanDates.txt call:

plot_eff.py --infilename=listOfScanDates.txt --vfatList=12,13 --latSig=39 --print

To calculate the efficiency using VFAT4 using background subtraction first call anaUltraLatency.py on each of the scandates given in listOfScanDates.txt and then call:

plot_eff.py --infilename=listOfScanDates.txt -v4 --bkgSub --print

The following tools exist to help you to analyze scans taken with xDAQ:

• anaXDAQLatency.py

See documentation written on the GEM DOC Twiki Page.

There are two tools for helping you to make arbitrary plots from python scan data:

• gemPlotter.py
• gemTreeDrawWrapper.py

The first tool is for plotting from multiple different scandates. The second tool is for making a given plot from a list of scandates, for each scandate.

The gemPlotter.py tool is for making plots of an observable stored in one of the TTree objects produced by the (ana-) ultra scan scripts vs an arbitrary indepdent variable specified by the user. Here each data point is from a different scandate. This is useful if you run mulitple scans in which only a single parameter is changed (e.g. applied high voltage, or VThreshold1) and you want to track the dependency on this parameter.

Each plot produced will be stored as an output *.png file. Additionally an output TFile will be produced which will contain each of the plots, stored as TGraph objects, and canvases produced.

The following table shows the mandatory inputs that must be supplied to execute the script:

Note for those anaType values which have the substring Ana in their names it is expected that the user has already run ana_scans.py on the corresponding scandate to produce the necessary input file for gemPlotter.py.

The following table shows the optional inputs that can be supplied when executing the script:

The format of this input file should follow the Three Column Format.

To automatically consider the last two weeks worth of s-curve scans, run the script specifying vt1bump option like this:

plotTimeSeries.py --vt1bump=10

resulting plots will be stored under

$ELOG_PATH/timeSeriesPlots/<chamber name>/vt1bumpX/

To make a 1D plot for a given strip of a given VFAT execute:

gemPlotter.py --infilename=<inputfilename> --anaType=<anaType> --branchName=<TBranch Name> --vfat=<VFAT No.> --strip=<Strip No.>

For example, to plot trimDAC vs. an Indep. Variable Name defined in listOfScanDates.txt for VFAT 12, strip number 49 execute:

gemPlotter.py -ilistOfScanDates.txt --anaType=trimAna --branchName=trimDAC --vfat=12 --strip=49

Additional VFATs could be plotted by either:

• Making successive calls of the above command and using the --rootOpt=UPDATE,
• Using the --vfatList argument instead of the --vfat argument, or
• Using the -a argument to make all VFATs.

To make a 1D plot for a given VFAT execute:

gemPlotter.py --infilename=<inputfilename> --anaType=<anaType> --branchName=<TBranch Name> --vfat=<VFAT No.> 

For example, to plot trimRange vs. an Indep. Variable Name defined in listOfScanDates.txt for VFAT 12 execute:

gemPlotter.py -ilistOfScanDates.txt --anaType=trimAna --branchName=trimRange --vfat=12

Note if TBranch Name is a strip level observable the data points (y-error bars) in the produced plot will represent the mean (standard deviation) from all of the VFAT's channels.

Additional VFATs could be plotted by either:

• Making successive calls of the above command and using the --rootOpt=UPDATE,
• Using the --vfatList argument instead of the --vfat argument, or
• Using the -a argument to make all VFATs.

To automatically extend this to all channels execute:

gemPlotterAllChannels.sh <InFile> <anaType> <branchName>

To make a 2D plot for a given VFAT execute:

gemPlotter.py --infilename=<inputfilename> --anaType=<anaType> --branchName=<TBranch Name> --vfat=<VFAT No.> --make2D

Here the output plot will be of the form "ROB Strip/VFAT Channel vs. Indep. Variable Name" with the z-axis storing the value of --branchName.

For example to plot trimDAC for "ROB Strip vs. Indep. Variable Name" wher For example to make a 2D plot with the z-axis as trimDAC and the Indep. Variable Name defined in listOfScanDates.txt for VFAT 12 execute:

gemPlotter.py -ilistOfScanDates.txt --anaType=trimAna --branchName=trimDAC --vfat=12 --make2D 

Additional VFATs could be plotted by either:

• Making successive calls of the above command and using the --rootOpt=UPDATE,
• Using the --vfatList argument instead of the --vfat argument, or
• Using the -a argument to make all VFATs.

The gemTreeDrawWrapper.py tool is for making a given 'Y vs. X' plot for each scandate of interest. Here Y and X are quantities stored in TBranches of one of the TTree objects procued by the (ana-) ultra scan scripts. This is designed to complement gemPlotter.py and should speed up plotting in general. This tool is essesntially a wrapper for the TTree::Draw() method. To make full use of this tool you should familiarize yourself with the TTree::Draw() documentation.

Additionally gemTreeDrawWrapper.py can also fit produced plots with a function defined at runtime through the command line arguments.

Each plot produced will be stored as an output *.png file. Additionally an output TFile will be produced which will contain each of the plots, stored as TGraph objects, canvases, and fits produced.

The following table shows the mandatory inputs that must be supplied to execute the script:

Note for those anaType values which have the substring Ana in their names it is expected that the user has already run ana_scans.py on the corresponding scandate to produce the necessary input file for gemTreeDrawWrapper.py.

The following table shows the optional inputs that can be supplied when executing the script:

The format of this input file should follow the Two Column Format.

For example to make a plot from a latency scan, Nhits vs. lat for VFAT12, use the following example:

gemTreeDrawWrapper.py -ilistOfScanDates_TreeDraw.txt --anaType=latency --summary --treeExpress="Nhits:lat" --treeDrawOpt=APE1 --treeSel="vfatN==12" --axisMaxY=1000 --axisMinX=39 --axisMaxX=49 --drawLeg

This will produce one Nhits vs. lat plot for VFAT12 for each (ChamberName,scandate) pair found in listOfScanDates_TreeDraw.txt. Additionally it will make one summary plot with a legend drawn which contains all of the produced plots.

For example to plot and fit an scurve from an scurve scan, Nhits vs vcal, for VFAT12 channel 45, use the following example:

gemTreeDrawWrapper.py -ilistOfScanDates_TreeDraw.txt --anaType=scurve --treeExpress="Nhits:vcal" --treeDrawOpt=APE1 --treeSel="vfatN==12 && vfatCH==45" --fitFunc="500*TMath::Erf((TMath::Max([2],x)-[0])/(TMath::Sqrt(2)*[1]))+500" --fitRange=70,150 --fitOpt="RM" --fitGuess=110,10,10

Here the fit that will be applied will be equivalent too:

myFunc = r.TF1(strName,"500*TMath::Erf((TMath::Max([2],x)-[0])/(TMath::Sqrt(2)*[1]))+500",70,150)
myFunc.SetParameter(0,110)
myFunc.SetParameter(0,10)
myFunc.SetParameter(0,10)

The fit option that will be used will be RM. This fit will be applied to the scurve generated from VFAT12 channel 45 for each (ChamberName,scandate) pair found in listOfScanDates_TreeDraw.txt.

The following tools exist for helping to understand scurve data:

1. gemSCurveAnaToolkit.py
2. plot_noise_vs_trim.py
3. plot_vfat_and_channel_Scurve.py
4. plot_vfat_summary.py
5. summary_plots.py

These tools can all by found in the macros/ subdirectory and are designed to be run on TFile objects containing the scurveFitTree TTree object (e.g. produced by anaUltraScurve.py). The first tool gemSCurveAnaToolkit.py is for plotting the same (vfat,channel/ROBstr) scurve from a list of scandates and it is described in a dedicated subsection below. The rest of the tools above are for making plots from a single input file; the plots made by tools 2-4 are:

• plot_noise_vs_trim.py: Plots a channel/strip's scurve width (e.g. noise) vs. trimDAC as a TH2D on a TCanvas,
• plot_vfat_and_channel_Scurve.py: Plots a channel/strip's scurve as a TH1D and its TF1 on a TCanvas, and
• plot_vfat_summary.py: Plots all scurves from a given VFAT as a TH2D on a TCanvas.

Tool 5 summary_plots.py produces the following plots from a single input file for a given VFAT depending on the command line argument supplied:

• Plot of channel/strip scurve mean as a TH1D,
• Plot of channel/strip scurve width as a TH1D,
• Plot of channel/strip scurve pedestal as a TH1D,
• Plot of Chi² of the channel/strip scurve fits as a TH1D,
• Plot of channel/strip scurve mean vs. scurve width as a TH2D, and
• Plot of channel/strip scurve width vs. trimDAC as a TH2D.

The command line options for tools 2-5 are:

Additionally tool 5 summary_plots.py has the following additional command line options:

Note that for tool 5 summary_plots.py you must supply at least one of these additional options {-a,-f,-x}.

The gemSCurveAnaToolkit.py tool is for plotting scurves and their fits from a given (vfat, vfatCH/ROBstr) from a list of scandates that correspond to TFile objects which contain the scurveFitTree TTree (e.g. files produced by anaUltraScurve.py). Each plot produced will be stored as an output *.png file. Additionally an output TFile will be produced which will contain each of the scurves and their fits.

The format of this input file should follow the Two Column Format.

To plot the scurves, and their fits, for VFAT0 channel 29 from a set of scandates defined in listOfScanDates_Scurve.txt taken by ultraScurve.py and analyzed with anaUltraScurve.py you would call:

gemSCurveAnaToolkit.py -ilistOfScanDates_Scurve.txt -v0 -s29 --anaType=scurveAna -c --summary --drawLeg

This will produce a *.png file for each of the scandates defined in listOfScanDates_Scurve.txt and one *.png file showing all the scurves with their fits drawn on it as a summary. Additionally an output TFile will be produced containing each of the scurves and their fits.

While gemTreeDrawWrapper.py and gemPlotter.py allow you to plot observables from multiple runs sometimes you are interested in seeing the results made from anaUltraScurve.py, from multiple scandates, on the same set of TCanvases. The tool plotSCurveFitResults.py allows you to do this. The tool will create five output *.png files and one TFile which stores relevant plots for each VFAT from each of the input scandates. These five *.png files are:

• scurveFitSummaryGridAllScandates.png, shows the fitSummary curves from all input scandates on one TCanvas in a 3-by-8 grid,
• scurveMeanGridAllScandates.png, shows the distribution of s-curve mean positions from each VFAT from all input scandates on one TCanvas in a 3-by-8 grid,
• scurveSigmaGridAllScandates.png, as scurveMeanGridAllScandates.png but for s-curve sigma,
• canvSCurveSigmaDetSumAllScandates.png, shows a summary distribution of s-curve sigma positions from all VFATs of the detector from all scandates on one TCanvas, and
• canvSCurveMeanDetSumAllScandates.png, as canvSCurveSigmaDetSumAllScandates.png but for s-curve mean.

The files will be found in $ELOG_PATH along with the output TFile, named scurveFitResultPlots.root.

The format of this input file should follow the Three Column Format. Note that here the Indep. Variable for each row will be used as the TLegend entry if the --drawLeg argument is supplied.

To plot results from a set of scandates defined in listOfScanDates_Scurve.txt taken by either ultraScurve.py or trimChamber.py and analyzed with anaUltraScurve.py you would call:

plotSCurveFitResults.py --anaType=scurveAna --drawLeg -i listOfScanDates_Scurve.txt --alphaLabels

This will produce the five *.png files mentioned above along with the output TFile.

timeHistoryAnalyzer.py is a tool that finds when a channel turns bad (see below for the available definitions), and possibly when it is recovered. It takes as input a set of files produced by plotTimeSeries.py, and the results are printed to the terminal.

The analysis proceeds in three steps, executed in the following order:

1. Bad scan removal: Scans that failed to produce consistent results are removed.
2. Range detection: The time evolution of each channel is searched for successive scans with consistent "bad" behavior (see below). A set of such scans for a given channel is called a (time) range. What kind of behavior is searched for is used-defined.
3. Analysis: The properties of "ranges" are computed and printed.

Scans that pass any the following cuts are removed:

• The average noise over the entire detector is lower than 0.1 fC (or --minScanAvgNoise). This cuts scans with no or very few channels responding.
• The fraction of masked channels is above 7% (or --maxScanMaskedFrac). This cuts e.g scans that produced no data and for which all fits failed.

Note that the options are named in the positive way, ie they tell which scans to keep.

The time evolution of each channel is searched for successive scans with consistent behavior. A set of such scans "bad" scans for a given channel is called a (time) range; the definition of bad is user defined (see below).

Range finding starts with a list of scans, where each scan is marked as "good" or "bad". The definition of "bad" depends on what's being searched for (and "good" is always defined as "not bad"). The start of a range is determined by:

• Starts with a "bad" scan (see below)
• The channel wasn't "bad" in the previous scan (e.g going good to bad)

Then the range continues and the end of the range is determined by 5 consecutive good scans appearing (option: --numEndScans). To prevent the printing of spurious ranges due to transient effects ranges with less than 4 "bad" scans in total are suppressed (option: --minBadScans). A "range" found by this algorithm can have include some "good" scans.

As a side-effect, channels with sparse "bad" behavior are also extracted. This can be controlled by tightening the cuts in the algorithm above.

Three definitions of "bad" are currently available:

• mask: the channel under consideration is masked
• maskReason: the channel under consideration has a non-zero maskReason
• zeroInputCap: the channel under consideration has an scurve width that is consistent with zero input capacitance (4.14E-02 < scurevWidth < 1.09E-01 fC). The precise values can be controlled using the --minNoise and --maxNoise options.

For every "range" found in each of the VFATs, the following properties are computed and printed in a table:

A summary table of initial maskReason vs VFAT is also printed at the end.

The examples below assume that you have analyzed S-curves using plotTimeSeries.py, and that the output is located at:

$ELOG_PATH/timeSeriesPlots/<chamber name>/vt1bumpX/

Note that the above structure is created automatically by plotTimeSeries.py.

The simplest possible call to timeHistoryAnalyzer.py is:

timeHistoryAnalyzer.py -i $ELOG_PATH/timeSeriesPlots/<chamber name>/vt1bumpX/

This will use the default range finder, maskReason, and settings. Depending on the detector and number of scans being analyzed, it may result in a lot of output being printed to the terminal. For every VFAT, you will get a table that looks like this:

The meaning of the column headers is explained above. Here's the information that we can extract from the table (take a look here first if you're not confident with the meaning of maskReason):

• Strip number 18 became hot between 2017.10.11.11.24 and 2017.10.13.12.53. In the same period of time, strip number 31 died.
• Strip number 91 became hot in July 2017; afterwards, it was also found to have a high noise. It was recovered in February 2018. The masked fraction at 47% indicates that during this period, about half the scans didn't result in the corresponding channel being masked.
• Strip number 93 was hot during two periods: from the end of March to the end of May 2017, and afterwards from the beginning of April 2017 to the beginning of February 2018. Since both ranges have similar properties and the masked fraction is low, the split in two is likely an accident.

The example above used the maskReason range finder. Let's try with zeroInputCap:

timeHistoryAnalyzer.py -i $ELOG_PATH/timeSeriesPlots/<chamber name>/vt1bumpX/ --ranges zeroInputCap

Note that --ranges zeroInputCap typically produces in a lot less output than the default.

At the end of its output, timeHistoryAnalyzer.py prints the following table (some lines were stripped for concision):

The first column is the VFAT number; the others correspond to the possible entries in maskReason.

The table counts how many times a given MaskReason appears in the "Initial maskReason" column of each per-VFAT tables. Indeed, if we look at VFAT 0 for the above example, we find:

The two entries in the DeadChannel column correspond to two ranges, that turn out to be from different strips (this may not be the case). Now VFAT 7:

We can see that the three entries in the DeadChannel column and the two in the HotChannel column come from the same ranges.

Note When using the --onlyCurrent option, there's only one range per channel, which makes the table easier to understand.

You may occasionally need to update the travis CI docker which checks the code quality or you may want to transfer a number of files corresponding to a series of scandates from the P5 machine to another area. The packageFiles4Docker.py tool enables you to do this. The output of packageFiles4Docker.py will be a *.tar file that:

• mimics the file structure of $DATA_PATH, and
• each of the input listOfScandates.txt files supplied at runtime, and
• a temorary chamberInfo.py file which can be placed in the docker for testing.

Please note that multiple --fileListX arguments can be supplied at runtime, but at least one must be supplied.

Each of the --fileListX arguments can be supplied with a listOfScanDates.txt file that follows either the Two Column Format or the Three Column Format.

To make a tarball of containing scurve scandates defined in listOfScanDates.txt for GEMINIm01L1 execute:

packageFiles4Docker.py --ignoreFailedReads --fileListScurve=$DATA_PATH/GEMINIm01L1/scurve/listOfScanDates.txt --tarBallName=GEMINIm01L1_scurves.tar --ztrim=4 --onlyRawData

In this case failed read errors in the tar command will be ignored and only the raw data, e.g. SCurveData.root files, will be stored in the tarball following the appropriate file structure.

It may be that eventually you will need to re-analyze a large portion of the calibration dataset. While this is expected to be rare it would be excessively time consuming to analyze the data by hand. This section details the tools that exist to assist you in this process. All tools below are designed to work with the lxplus batch submission system based on LSF. Please note CERN IT plans to eventually transition from LSF to HTCondor. When this occurs these tools will need to be migrated to the new system. Instructions for doing so are available here.

This tool will allow you to re-analyze the scurve data in a straight forward way without the time consuming process of launching it by hand.

The following table shows the mandatory inputs:

While the following table shows the optional additional inputs:

Finally clusterAnaScurve.py can also be passed the cut values used in assigning a maskReason described at Providing Cuts for maskReason at Runtime.

Before you start due to space limitations on AFS it is strongly recommended that your $DATA_PATH variable on lxplus point to the work area rather than the user area, e.g.:

export DATA_PATH=/afs/cern.ch/work/<first-letter-of-your-username>/<your-user-name>/<somepath>

In your work area you can have up to 100GB of space. If this is your first time using lxplus you may want to increase your storage quota by following instructions here.

Now connect to the P5 dqm machine. Then after setting up the env execute if you are intereted in a chamber ChamberName execute:

cd $HOME
plotTimeSeries.py --listOfScanDatesOnly --startDate=2017.01.01
packageFiles4Docker.py --ignoreFailedReads --fileListScurve=/gemdata/<ChamberName>/scurve/listOfScanDates.txt --tarBallName=<ChamberName>_scurves.tar --ztrim=4 --onlyRawData

Then connect to lxplus. Checkout the repository if you have not done so already. Then after setting up the env execute:

cd $DATA_PATH
scp <your-user-name>@cmsusr.cms:/nfshome0/<your-user-name>/<ChamberName>_scurves.tar .
tar -xf <ChamberName>_scurves.tar
mv gemdata/<ChamberName> .
clusterAnaScurve.py -i <ChamberName>/scurve/listOfScanDates.txt --anaType=scurve -f -q 1nh

It may take some time to finish the job submission. Please pay attention to the output at the end of the clusterAnaScurve.py command as it provodes helpful information for managing jobs and undersanding what comes next. Once your jobs are complete you should check that they all finished successfully. One way to do this is to check if any of them exited with status Exited and check for the exit code. To do this execute:

grep -R "exit code" <ChamberName>/scurve/*/stdout/jobOut.txt --color 

This will print a single line from all files where the string exit code appears. For example:

% grep -R "exit code" GEMINIm01L1/scurve/*/stdout/jobOut.txt --color 
GEMINIm01L1/scurve/2017.04.10.20.33/stdout/jobOut.txt:Exited with exit code 255.
GEMINIm01L1/scurve/2017.04.26.12.25/stdout/jobOut.txt:Exited with exit code 255.
GEMINIm01L1/scurve/2017.04.27.13.27/stdout/jobOut.txt:Exited with exit code 255.
GEMINIm01L1/scurve/2017.06.07.12.17/stdout/jobOut.txt:Exited with exit code 255.
GEMINIm01L1/scurve/2017.07.18.11.09/stdout/jobOut.txt:Exited with exit code 255.
GEMINIm01L1/scurve/2017.07.18.18.34/stdout/jobOut.txt:Exited with exit code 255.

For those lines that appear in the grep output command you will need to check the standard err of the job which can be found in:

<ChamberName>/scurve/<scandate>/stderr/jobErr.txt

Note since some scans at P5 may have failed to complete successfully some jobs may intrinsically fail and be non-recoverable. If you have questions about a particular job you can try to search in the e-log around the scandate in time to see if anything occurred around this time that might cause problems for the scan. If you would like to re-analyze a failed job you can do so by calling:

source $DATA_PATH/<ChamberName>/scurve/<scandate>/clusterJob.sh

If a large number of jobs have failed you should spend some time trying to understand why, and then re-submit to the cluster, rather than attempting to analyze them all by hand.

Finally after you are satisfied that all the jobs that could complete successfully have completed you can:

1. re-package the re-analyzed data into a tarball, and/or
2. create time series plots to summarize the entire dataset.

For case 1, re-packaging the re-analyzed files into a tarball, execute:

packageFiles4Docker.py --ignoreFailedReads --fileListScurve=<ChamberName>/scurve/listOfScanDates.txt --tarBallName=<ChamberName>_scurves_reanalyzed.tar --ztrim=4
mv <ChamberName>_scurves_reanalyzed.tar $HOME/public
chmod 755 $HOME/public/<ChamberName>_scurves_reanalyzed.tar
echo $HOME/public/<ChamberName>_scurves_reanalyzed.tar

Then provide the terminal output of this last command to one of the GEM DAQ Experts for mass-storage.

For case 2, create time series plots to summarize the entire dataset, execute:

<editor of your choice> $VIRTUAL_ENV/lib/python*/site-packages/gempython/gemplotting/mapping/chamberInfo.py

And ensure the only uncommented entries of the chamber_config dictionary match the set of ChamberName's that you have submitted jobs for. Then execute:

plotTimeSeries.py --startDate=2017.01.01 --anaType=scurve

Please note the above command may take some time to process depending on the number of detectors worth of data you are trying to analyze. Then a series of output *.png and *.root files will be found at:

$ELOG_PATH/timeSeriesPlots/<ChamberName>/vt1bump0/

If you would prefer to analyze ChamberName's one at a time, or to have an output *.png file for each VFAT, you can produce time series plots individually by executing the gemPlotter.py commands provided at the end of the clusterAnaScurve.py output. This might be preferred as when analyzing a large period of time the 3-by-8 grid plots that plotTimeSeries.py will produce for you may be hard to read. In either case gemPlotter.py or plotTimeSeries.py will produce a TFile for you in which the plots at the per VFAT level are stored for you to later investigate.

If you encounter issues in this procedure please spend some time trying to figure out what wrong on your side first. If after studying the documentation and reviewing the commands you have exeuted you still do not understand the failure please ask on the Software channel of the CMS GEM Ops Mattermost team or submit an issue to the github page.