Add support for cross-section scaling via SampleHandler about xaodanahelpers HOT 10 CLOSED

Hi @fizisist -- any ideas about how we should generate our own cross-sections file? It seems like we would have to do it by hand. Would we look up all the values using AMI?

from xaodanahelpers.

Hi @kratsg,
There's an interface to do this programmatically, but often the cross sections are not exactly correct.
In any case, here are some examples for extracting this information:

https://ami.in2p3.fr/pyAMI/command_line.html#retrieving-dataset-information

I have an old script I used to use with an old version of AMI:

printf "%70s %6s %10s %10s %40s %14s %18s %20s %10s %12s \n" "Dataset" "N Files" "N Events" "Xsect" "PDF" "Release" "Geometry" "Conditions" "DB Release" "Physics"

for ds in ${list}; do
  dsEVNT=`amiListDatasetProvenance ${ds} | grep EVNT | awk '{print $1}'`
  dsSIM=`amiListDatasetProvenance ${ds} | grep HITS | awk '{print $1}'`
  amiGetDatasetInfo -logicalDatasetName=${dsEVNT} >> info.txt
  amiGetDatasetInfo -logicalDatasetName=${dsSIM} >> info.txt
  amiGetDatasetInfo -logicalDatasetName=${ds} >> info.txt
  awk -F " = " -f ~/scripts/AMIDatasetSummary.awk info.txt
done

where the awk program file looks like:

/logicalDatasetName/ { ds = $2 }
/nFiles/ { nfiles = $2 }
/totalEvents/ { nevents = $2 }
/TransformationPackage/ { release = $2 }
/version/ { config = $2 }
/geometryVersion/ { geo = $2 }
/conditionsTag/ { cond = $2 }
/DBRelease/ { db = $2 }
/crossSection_mean/ { xsect = $2 }
/GenFiltEff/ { eff = $2 }
/PDF/ { pdf = $2 }
/physicsList/ { physlist = $2 }

END {
  printf ("%70s %6d %10d %2.6e %40s %14s %18s %20s %10s %12s \n", ds, nfiles, nevents, xsect, pdf, release, geo, cond, db, physlist);
}

It should be more or less the same now, but the pyami commands and where they get placed in the output info file will be different.

Cheers,
David

from xaodanahelpers.

Thanks @fizisist . I talked with Jamie as well to see what he does. I'll add this feature in somehow.

#!/bin/bash

localSetupPyAMI

mv sample_metadata.dat sample_metadata.old
for x in $(cat ../data/GridSamples | awk '{printf $2"\n";}'); do

  ami show dataset info $x > temp

  nevts=$( cat temp | grep totalEvents         | awk '{printf $3;}')
  xs=$(    cat temp | grep approx_crossSection | awk '{printf $3;}')
  filter=$(cat temp | grep GenFiltEff          | awk '{printf $3;}')
  dsid=$(  cat temp | grep datasetNumber       | awk '{printf $3;}')
  dsname=$(cat temp | grep logicalDatasetName  | sed "s/.*$dsid.//" | sed "s/.merge.*//")

  [ "$xs" == "" ]     && xs=XS
  [ "$filter" == "" ] && filter=1.0

  echo NAME $dsid $dsname $nevts $xs $filter | tee -a sample_metadata.dat

done

column -t sample_metadata.dat > temp
mv temp sample_metadata.dat

from xaodanahelpers.

Bringing this back up.

It would be good if we have unified way of getting the MC weights.

Multijets and Dijet have a hack, where we are reading a separate text file in our dedicated algorithm c++ class and calculate the weights by hand .

What are others doing ? How should we proceed here ?

from xaodanahelpers.

Yeah, so here's what I decided on -- algorithms should have the job of appropriately applying event-by-event weights no matter what. At the very last steps for doing optimizations, event counts, plots -- one should apply the sample-by-sample weights. Part of this relies on cutbookkeepers being completely fixed (I do not know if it is) and I would rather not strictly rely on it for right now.

The readSusyMeta is our best solution and simply requires us to always have the updated files from SUSYTools (who does the work of collecting all the information for us).

from xaodanahelpers.

How well does the SUSY meta-data file work with private samples stored in a file list?

I have a large set of private samples for testingm all with the same run number. My current approach is to store the xsec/number of events in a .config file, similar to ProofAna. You can find an example on uct3:

/share/home/kkrizka/xAH/filelists/MC15.999980.MGPy8EG_dmA_dijetgamma_mR300_mDM10000_gSM0p16_gDM1p50.txt
/share/home/kkrizka/xAH/filelists/MC15.999980.MGPy8EG_dmA_dijetgamma_mR300_mDM10000_gSM0p16_gDM1p50.config

A modified xAH_run.py then sets the same MetaData properties as the SH AMI support (https://twiki.cern.ch/twiki/bin/view/AtlasProtected/SampleHandler#Querying_Meta_Data_from_AMI, which I use for the official samples) and as the SH SUSY meta-data loading tool.

from xaodanahelpers.

The SUSYMetaData approach basically loads all the files based on DIDs. It's commented out here:

https://github.com/UCATLAS/xAODAnaHelpers/blob/master/scripts/xAH_run.py#L335-L337

A working example is here: https://github.com/kratsg/TheAccountant/blob/master/scripts/CookTheBooks.py#L372-L374 with data here: https://github.com/kratsg/TheAccountant/tree/master/data/metadata

and I grab the files doing something like

cd TheAccountant/data/metadata
svn export --force svn+ssh://[email protected]/reps/atlasoff/PhysicsAnalysis/SUSYPhys/SUSYTools/trunk/data/susy_crosssections_13TeV.txt
svn export --force svn+ssh://[email protected]/reps/atlasoff/PhysicsAnalysis/SUSYPhys/SUSYTools/trunk/data/mc15_13TeV/
cd -

for normal derivations.

from xaodanahelpers.

ok got this working now.

Quick question, is this expected to modify the weight in returned by mcEventWeight() in EventInfo ?

or is the only side effect that
wk()->metaData()->castDouble(SH::MetaFields::crossSection ,1);
wk()->metaData()->castDouble(SH::MetaFields::filterEfficiency,1);

are filled ?

from xaodanahelpers.

It's only filling in metaData
On Mon, Nov 2, 2015 at 18:11 johnda102 [email protected] wrote:

ok got this working now.

Quick question, is this expected to modify the weight in returned by
mcEventWeight() in EventInfo ?

or is the only side effect that
wk()->metaData()->castDouble(SH::MetaFields::crossSection ,1);
wk()->metaData()->castDouble(SH::MetaFields::filterEfficiency,1);

are filled ?

—
Reply to this email directly or view it on GitHub
#19 (comment)
.

from xaodanahelpers.

this is now workng. Closing.

from xaodanahelpers.