cms-sw / cmsdist Goto Github PK

Please replace frontier_client 2.8.9 ASAP with 2.8.10. 2.8.9 crashes with a segmentation fault any time data is requested more than 5 minutes after the first data is requested.

This is for at least 7_1_X. I don't know if 7_0_X has been updated to 2.8.9 yet.

The branch:

https://github.com/giamman/cmssw/tree/pileup13tev

requires new files in:

/afs/cern.ch/cms/data/CMSSW/FastSimulation/PileUpProducer/data

These files are in:

/afs/cern.ch/user/g/giamman/scratch0/fastsim/minbias

we should add these and remove the 14T ones for pre6.

@giamman @davidlt

CMSSW_7_1_X needs to be switched to CLHEP 2_1_4_1 to be a starting point for making a patch for thread safe random generation. In GEANT4_10 branch this switch was already done (pull request 298).

New xml files should be updated in the /data externals in these two packages
RecoParticleFlow/PFTracking
RecoEgamma/ElectronIdentification

See details in
cms-sw/cmssw#1801

@lgray

2nd patch to Geant4 10.0 is available at url:
http://geant4.web.cern.ch/geant4/support/download.shtml

As was agreed at last OPR and SIM meetings this patch should be first integrated to special IB 7_1_0_g410p02 to be compared with 7_1_0.

10.0p02 should be built in the same way as 10.0p01 in sequential mode and with the same other compile options.

For 7_2_X_GEANT10 it should be built in MT mode with modifications of cmake scripts as was done when 10.0p01 was built.

Please update CMSSW to use new frontier_client release 2.8.9. It includes a new "forever" time-to-live (ttl) feature that is needed for the new CMS Conditions data reorganization . The release notes and distribution are at http://frontier.cern.ch/dist/.

I don't know which CMSSW release series it should go into. I understand that 7_0_0 is closed, so it can go into the next open series. I will soon be submitting a patch to coral that will also be needed in order for CMSSW to use the new forever ttl feature.

Hi, i'd like to add these two boost libraries as external tools ( related code will be in 7_0_0 )

boost_iostreams
boost_serialization

Thanks

Giacomo

In the case of standalone Geant4 usage of static build of Geant4 and its application bring benifit in CPU about 20%.

For CMSSW GEN-SIM step Geant4 is used by following list of sub-packages:

SimG4CMS/Calo/
SimG4CMS/FP420/
SimG4CMS/Forward/
SimG4CMS/Muon/
SimG4CMS/Tracker/
SimG4Core/Application/
SimG4Core/CustomPhysics/
SimG4Core/Generators/
SimG4Core/Geometry/
SimG4Core/HelpfulWatchers/
SimG4Core/MagneticField/
SimG4Core/Notification/
SimG4Core/Physics/
SimG4Core/PhysicsLists/
SimG4Core/SensitiveDetector/
SimRomanPot/SimFP420/

For feasibility check we need to organise one "global" sub-package, let say SimG4Core/Simulation which will include all files from these 16 sub-packes.

Hi,

This library is now used internally in EventFilter/Utlities. DQM group has also expressed interest in using it (to produce json files for DQM stream merging), so I suggest to include this library as an external dependency.

Library code is located here:
http://sourceforge.net/projects/jsoncpp/files/
EventFilter/Utilities includes v0.5.0 which is a latest stable version.

Unfortunately, the source comes only with MSVC build files, so I had to create a simple make file to produce a shared library:
/afs/cern.ch/user/s/smorovic/public/jsoncpp/Makefile
(also I copied the source tar gz here)

The rpm spec should not be complicated to do. It only needs to take what is in "include", as well as a library/symlink created by make in "lib" directory.

Regards,
Srecko

I would like to ask to build and distribute "eigen" as external in CMSSW

site
http://eigen.tuxfamily.org/index.php?title=Main_Page
version tested in CMSSW_7_2_X
http://bitbucket.org/eigen/eigen/get/3.2.2.tar.gz

eigen is header only

A new shower HF library file needs to be added to HCAL data:

/afs/cern.ch/user/k/kheyn/public/For_Salavat/HFShowerLibrary_oldpmt_eta4_16en.root

This SL will be customized for 2011-2012 reprocessing.

Other two existing files with SL should not be removed.

From @VinInn:

pb-d-128-141-131-26:~/CMSSW_7_0_0_pre0/src innocent$ git cms-addpkg DataFormats/Math
No release tags specified, using default CMSSW_7_0_0_pre0.
You are on branch CMSSW_7_0_X
Checking out DataFormats/Math in tag CMSSW_7_0_0_pre0.
Your reference git repository does not seem to exist, would you like to create it? [ y / N / ? ] yCloning into bare repository '/Users/innocent/.cmsgit-cache'...
fatal: unable to access 'https://github.com/cms-sw/cmssw.git/': SSL certificate problem: unable to get local issuer certificate

@davidlt Are you sure we have a working curl+openssl on osx? I vaguely remember similar issues with standalone curl, could you check?

Hi.
This is a follow-up on the request for changing the bpmpd code to external. The change was decided in the RECO/AT meeting of December 5, 2013.
https://indico.cern.ch/conferenceDisplay.py?confId=271466#2013-12-05

The technical details of the package were discussed here:
https://hypernews.cern.ch/HyperNews/CMS/get/swDevelopment/2915.html

Thanks for any help.

Hi, Folks,

There is a new version of the "fastjet contrib" external tool (1.014) with new items we would like to use in JME. The instructions to add it are :

cmsrel CMSSW_7_2_0_pre6
cd CMSSW_7_2_0_pre6/src/
cmsenv
wget http://fastjet.hepforge.org/contrib/downloads/fjcontrib-1.014.tar.gz
tar -zxvf fjcontrib-1.014.tar.gz
cd fjcontrib-1.014
setenv FASTJET_BASE scramv1 tool tag fastjet FASTJET_BASE
./configure --fastjet-config=$FASTJET_BASE/bin/fastjet-config --prefix=$PWD CXXFLAGS="-I$FASTJET_BASE/include -I$FASTJET_BASE/tools"
make
make check
make install
make fragile-shared
make fragile-shared-install

Cheers,
Sal, Andreas, Nhan, Raffaele, Jim

A new HF shower library has been created to be used with 7_1_0_pre8. It is available at

/afs/cern.ch/user/k/kheyn/public/For_Salavat/HFShowerLibrary_npmt_eta4_16en.root

and should be accessible in /afs/cern.ch/cms/data/CMSSW/SimG4CMS/Calo/data

Old shower libraries should not be removed.

I would like to create an external for CalibTracker/SiPixelESProducers/data as proposed ny slava77 in the pull request discussion:
cms-sw/cmssw#3236

Cheers,
Pal

Necessary for: cms-sw/cmssw#2242

The updated file is already in the central afs area:
http://cmsdoc.cern.ch/cms/data/CMSSW/RecoEgamma/ElectronIdentification/data/TMVA_BDTSoftElectrons_27Jan2014.weights.xml

Please update the soft-electron MVA weights file, as found in cms-sw/cmssw#2717

Thanks!

Geant4 10 has been merged to the main development branch using sequential build of geant4 10.0p01. For multi-threaded developments we need to return back previous multi-threaded build of Geant4 10.0p01 with cms private patch of build options for the GEANT10 branch.

We need to have support for central man-pages, in particular for cms-git-utils, but also for scram and a few other tools we have. @davidlt @smuzaffar ideas where to put them? shall we have an extra line in cmsset_default.{sh,csh} which adds MANPATH=%instroot/common/man:$MANPATH?

@mkirsano

I created a spec file to build Rivet 2.0.0 following the integration of #374.

I am experiencing a weird error during the configuration of Rivet:

checking fastjet/TrackJetPlugin.hh usability... yes
checking fastjet/TrackJetPlugin.hh presence... yes
checking for fastjet/TrackJetPlugin.hh... yes
configure: Found yaml-cpp library at /build/degano/rivet_20/a/slc6_amd64_gcc481/external/yaml-cpp/0.5.1/lib/libyaml-cpp.so
configure: Found yaml-cpp header directory at /build/degano/rivet_20/a/slc6_amd64_gcc481/external/yaml-cpp/0.5.1/include
checking yaml-cpp/node.h usability... no
checking yaml-cpp/node.h presence... no
checking for yaml-cpp/node.h... no
checking yaml-cpp/node/node.h usability... no
checking yaml-cpp/node/node.h presence... no
checking for yaml-cpp/node/node.h... no
configure: error: yaml-cpp API version could not be determined

Although the header yaml-cpp/node/node.h is actually there:

ll /build/degano/rivet_20/a/slc6_amd64_gcc481/external/yaml-cpp/0.5.1/include/yaml-cpp/node/node.h 
-rw-r--r--. 1 degano zh 3335 Apr 14  2013 /build/degano/rivet_20/a/slc6_amd64_gcc481/external/yaml-cpp/0.5.1/include/yaml-cpp/node/node.h

Do you happen to know what might be the reason of the failure?

Please upgrade doxygen in next CMSSW release.
Doxygen source: http://www.stack.nl/~dimitri/doxygen/download.html

Full migration to the version 10.0 includes updates of CLHEP, data files and source code itself. The patch of cmake/gmake configure options prepared for previous version of Geant4 (pull request 265) needs to be applied on top of Geant4.

CLHEP:
http://proj-clhep.web.cern.ch/proj-clhep/DISTRIBUTION/tarFiles/clhep-2.1.4.1.tgz

Geant4 download page url:
http://geant4.web.cern.ch/geant4/support/download.shtml

Source:
geant4.10.00.tar.gz

Updated datasets:
G4NDL.4.4.tar.gz
G4EMLOW.6.35.tar.gz
G4PhotonEvaporation.3.0.tar.gz
G4RadioactiveDecay.4.0.tar.gz
G4NEUTRONXS.1.4.tar.gz
G4ABLA.3.0.tar.gz

Hi,

Would it be possible to include the FastJet Contrib package in 5_3_X releases? It has already been added to 7_1_X releases (#369 and #393). This would simplify development and testing of new tools as well as backporting certain features from more recent releases.

Thank you,
Dinko

Dear experts;
we (@yslai @mtonjes @mandrenguyen) are working on preparing 5_3_X for new HI jet reconstruction, which requires some calibration files in txt format.
Based on discussions with @slava77 we need the following files :

/afs/cern.ch/user/y/yilmaz/public/ue/ue_calibrations_calo_data.txt
/afs/cern.ch/user/y/yilmaz/public/ue/ue_calibrations_calo_mc.txt
/afs/cern.ch/user/y/yilmaz/public/ue/ue_calibrations_pf_data.txt
/afs/cern.ch/user/y/yilmaz/public/ue/ue_calibrations_pf_mc.txt

to be copied into:

http://cmsdoc.cern.ch/cms/data/CMSSW/RecoHI/HiJetAlgos/data/

in order to use them in the upcoming 5_3_X release as externals. Can you please copy them?

Thanks.

A new patch to Geant4 10.0 is available which include mainly fixes for MT mode. We need first to validate the patch in GEANT10 branch. The patch is available via url: http://cern.ch/geant4/support/download.shtml

Build method and options should be the same as in the case of previous version Geant4 10.0p02.

According to the plan of Geant4 10.0p01 integration a special build needs to be created on top of CMSSW_7_1_0_pre4, which includes all modifications available at the GEANT10 branch. This build will be used for RelVal validation versus pre4.

Practically it will be a merge of GEANT10 to 7_1_0_pre4.

For this special build will be optimal to recompile Geant4 10.0p01 in sequential mode, do not apply CMS private patch, and use Geant4 data files from GEANT10 branch.

GEANT10 branch should be left unchanged, it will be used for further development of MT variant of CMSSW simulation.

When update any package and trying to build locally a test therê is a problem whihc is seen like:

/afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc5_amd64_gcc481/external/gcc/4.8.1/bin/../lib/gcc/x86_64-redhat-linux-gnu/4.8.1/../../../../x86_64-redhat-linux-gnu/bin/ld: 
_ZN14G4GeomSplitterI8G4LVDataE6offsetE: TLS reference in /afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc5_amd64_gcc481/cms/cmssw/CMSSW_7_0_GEANT10_X_2013-11-06-1400/external/slc5_amd64_gcc481/lib/libG4digits_hits.so
mismatches non-TLS reference in
tmp/slc5_amd64_gcc481/src/Alignment/LaserAlignmentSimulation/src/AlignmentLaserAlignmentSimulation/LaserSteppingAction.o

The problem is connected with the fact that geant4 9.6ref10 was build with the flag -DGEANT4_BUILD_MULTITHREADED=ON while simple "scram b -j8" does not know anything about MULTITHREADED. How to tell scram about this flag?

Hi all,

to provide XML output for the condDBv2 conditions, we need an update of boost (presently 1.51.0) to version 1.57.0 (latest). In a private build, I've verified that the new version fixes the issue which exists in 1.51.0. Initially we would need that in 7.3.x though once validated it needs to be back ported to the 7.2.x releases for use in online environments.

Please let me know if you need further information.

Thanks,
cheers, andreas

I have developed a patch for coral that I need included in CMSSW. I don't see a way to attach a file to a github issue, so find the patch on lxplus at
~/public/coral-CORAL_2_3_21-forever-ttl.patch

I have tested this on SLC5 with SCRAM_ARCH=slc5_amd64_gcc481 against release candidate CORAL_2_3_21-cms10 and CMSSW_7_0_0_pre11, and on SLC6 with SCRAM_ARCH=slc6_amd64_gcc481 against CORAL_2_3_21-cms9 and CMSSW_7_0_0_pre11.

Depends on #305.

Next I will be submitting this patch to coral for inclusion in a new release of theirs, so in future builds this patch will be able to be removed.

Seems that current GIT packages contains non-reallocated files:

bin/git-cvsserver
lib64/perl5/5.8.8/x86_64-linux-thread-multi/perllocal.pod
lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Git/.packlist
libexec/git-core/git-add--interactive
libexec/git-core/git-archimport
libexec/git-core/git-cvsexportcommit
libexec/git-core/git-cvsimport
libexec/git-core/git-cvsserver
libexec/git-core/git-difftool
libexec/git-core/git-instaweb
libexec/git-core/git-relink
libexec/git-core/git-send-email
libexec/git-core/git-svn
lib/perl5/site_perl/5.8.8/Git/I18N.pm
share/gitweb/gitweb.cgi

1st patch to Geant4 10.0 is available at url:
http://geant4.web.cern.ch/geant4/support/download.shtml

As was agreed at last OPR meeting this patch should be first integrated to 7_1_0_GEANT10 branch.

In the patch both make and cmake compile options were changed but likely a small CMS private patch similar to what was done for 10.0 is needed for 10.0p01

During winter break number of SIM production jobs were lost due to the new Geant4 bug. The fix should be applied for CMSSW_7_0_X, CMSSW_6_2_X, and CMSSW_6_2_X_SLHC.

The patch is very simple and should not affect any CMS object but only fix crashes in production. It can be accessed via url: https://vnivanch.web.cern.ch/vnivanch/geant4-9.6p02-cms01.patch

Hi,
looking at the IBs, the IB/CMSSW_7_3_X/debug branch is failing because it is missing the latest Pythis8 updates.

Can you set up the automatic synchronisation of IB/CMSSW_7_3_X/debug with IB/CMSSW_7_3_X/stable ?

@covarell is the new generator convener

(Already started a conversation with David A about this via email, but filing here)

We'd like the Xrootd external updated to 3.3.3-rc1 for the CMSSW_7_0_X series. This is to allow me to enable the usage of the new Xrootd client from the Xrootd 3.3.X series.

RecoEgamma/EgammaIsolationAlgos package is ICE'ing the Clang and RecoVertex/AdaptiveVertexFit is capturing non-POD VLAs in lambda. The latter landed in Clang after 3.5.0 release.

Both of these are resolved in newer Clang versions.

The following seems to resolve it.

diff --git a/llvm.spec b/llvm.spec
index dc1532e..e44600a 100644
--- a/llvm.spec
+++ b/llvm.spec
@@ -4,11 +4,12 @@
 BuildRequires: python
 Requires: gcc

-%define llvmCommit f55c17bc339
-%define clangCommit 656aa83af0b
-%define branch cms/v%realversion
-Source0: git+https://github.com/cms-externals/llvm.git?obj=%{branch}/%{llvmCommit}&export=llvm-%{realversion}-%{llvmCommit}&module=llvm-%realversion-%llvmCommit&output=/llvm-%{real
-Source1: git+https://github.com/cms-externals/clang.git?obj=%{branch}/%{clangCommit}&export=clang-%{realversion}-%{clangCommit}&module=clang-%realversion-%clangCommit&output=/clang
+%define llvmCommit e1e1392e13ef3f5e97fbdd15ca8d60d63f69a862
+%define clangCommit 097068701d2c7fff58866059b1e3ebcac2728f7a
+%define llvmBranch cms/e1e1392
+%define clangBranch cms/9443d04
+Source0: git+https://github.com/davidlt/llvm.git?obj=%{llvmBranch}/%{llvmCommit}&export=llvm-%{realversion}-%{llvmCommit}&module=llvm-%realversion-%llvmCommit&output=/llvm-%{realve
+Source1: git+https://github.com/davidlt/clang.git?obj=%{clangBranch}/%{clangCommit}&export=clang-%{realversion}-%{clangCommit}&module=clang-%realversion-%clangCommit&output=/clang-

 # Still need forward porting.
 %define keep_archives true

LLVM upstream commit e1e1392e13ef3f5e97fbdd15ca8d60d63f69a862
Clang upstream commit 9443d047e89e716928fd87d277fca673afc2ec43

This is a working item for me to find and propose working LLVM and Clang commits for the update.

Hi,

If it is possible, I'd like to include this external package for fast multi-dimensional spline fitting in CMSSW. I need it to provide cross section and shape-morphing information for a combined HIG-SMP fit (Higgs + Multiboson).
I've been able to get it to compile within the standard CMSSW environment, but can be a little painful to setup since it uses cmake and it would be nice to have it properly integrated and recognized as a scram tool.

You can find the source code in the pre-print tarball at the arxiv:
http://arxiv.org/abs/1301.2184

Its dependencies are:
CMake (>= 2.8): http://www.cmake.org
cfitsio (>= 3.1): http://heasarc.gsfc.nasa.gov/fitsio/
gsl (>= 1.14): http://www.gnu.org/software/gsl/
SuiteSparse (>= 3.5): http://www.cise.ufl.edu/research/sparse/
PyFITS (>= 3.1): http://www.stsci.edu/institute/software_hardware/pyfits
Numpy (>= 1.3): http://www.numpy.org
Python (>= 2.5): http://www.python.org

Finally it needs an optimized BLAS and LAPACK.
The authors recommend: http://www.tacc.utexas.edu/tacc-projects/gotoblas2

For configuration details there's a thorough description in the source tarball under photospline/README.txt.

Please let me know if I need to edit any packages to compile with scram or if there is anything else I can do help this along.

Thanks!

Please update CMSSW to frontier_client 2.8.11 from http://frontier.cern.ch/dist.

This fixes a fatal, obscure error ("error -5: unknown error") when more than 1024 file descriptors are open.

Hi,
would it be possible to install the cms+cmssw-debug+CMSSW_* package for the CMSSW_7_3_DEBUG_X integration builds ?

.A

As per @slava77 suggestion, we are requesting an external package to store the text files now contained at:

https://github.com/dmitrijus/cmssw/tree/neural-corrections/RecoLocalCalo/EcalDeadChannelRecoveryAlgos/data/NNWeights

these are the weights of neural networks used for recovery of isolated single channels in Ecal

Report and details are here: https://hypernews.cern.ch/HyperNews/CMS/get/swDevelopment/3065/1/1/2.html

The dcap we provide is minimal, but is interfering with other utilities (GRID software). We need to provide dcap with same behavior/ABI compatible to the system one.

I already identified the change needed.

This is a working item for myself. It should be done for CMSSW_7_3_0.

Hello, I would like to propose inclusion of "FastJet Contrib" version 1.008 as an external tool in CMSSW_7_0_X. This is a set of 3rd party extensions to the FastJet package, managed by the authors of FastJet. It will be used for calculation of Njettiness, etc during jet reconstruction. More information is available here: http://fastjet.hepforge.org/contrib/

Hey,
would it be possible to add a section like:

<runtime name="PATH"    value="$PROTOBUF_BASE/bin"/>

to protobuf-toolfile.spec so that, after setting the tool via scram, the protoc executable is automatically found in the PATH?

Thanks,
Marco.

Necessary for:
cms-sw/cmssw#3690

The 9 files are already in the dedicated afs area:
http://cmsdoc.cern.ch/cms/data/CMSSW/RecoParticleFlow/PFTracking/data/MVA_BDTTrackDrivenSeed_cat1.xml
http://cmsdoc.cern.ch/cms/data/CMSSW/RecoParticleFlow/PFTracking/data/MVA_BDTTrackDrivenSeed_cat2.xml
http://cmsdoc.cern.ch/cms/data/CMSSW/RecoParticleFlow/PFTracking/data/MVA_BDTTrackDrivenSeed_cat3.xml
http://cmsdoc.cern.ch/cms/data/CMSSW/RecoParticleFlow/PFTracking/data/MVA_BDTTrackDrivenSeed_cat4.xml
http://cmsdoc.cern.ch/cms/data/CMSSW/RecoParticleFlow/PFTracking/data/MVA_BDTTrackDrivenSeed_cat5.xml
http://cmsdoc.cern.ch/cms/data/CMSSW/RecoParticleFlow/PFTracking/data/MVA_BDTTrackDrivenSeed_cat6.xml
http://cmsdoc.cern.ch/cms/data/CMSSW/RecoParticleFlow/PFTracking/data/MVA_BDTTrackDrivenSeed_cat7.xml
http://cmsdoc.cern.ch/cms/data/CMSSW/RecoParticleFlow/PFTracking/data/MVA_BDTTrackDrivenSeed_cat8.xml
http://cmsdoc.cern.ch/cms/data/CMSSW/RecoParticleFlow/PFTracking/data/MVA_BDTTrackDrivenSeed_cat9.xml

Would it be possible to create a new IB for deployment of the new Geant4 10.0.

The recent reference version is available via url:
http::/cern.ch/vnivanch/geant4.9.6ref10.tar.gz

possible cmake options to build this version:
cmake -DCMAKE_INSTALL_PREFIX=${G4 build dir}
-DGEANT4_USE_GDML=ON
-DGEANT4_INSTALL_DATADIR=${pass to G4 data}
-DXERCESC_ROOT_DIR=${pass to XERCESC}
-DGEANT4_INSTALL_EXAMPLES=OFF
-DGEANT4_ENABLE_TESTING=OFF
-DGEANT4_USE_SYSTEM_CLHEP=OFF
-DBUILD_SHARED_LIBS=ON
-DBUILD_STATIC_LIBS=OFF
-DGEANT4_BUILD_MULTITHREADED=ON
${G4INSTALL}

It is not needed to change data files and CLHEP version for the time being - current one can be used.

Please do no disable perl support when compiling git, as it disables git add --interactive and other stuff. Perl is a system requirement for CMSSW installations so you can expect to have it there. This is a showstopper for CMSSW_6_2_0.

Would it be possible to have the LLVM external install its Python bindings for clang (or as a new external)? i.e. these ones:

https://github.com/llvm-mirror/clang/tree/master/bindings/python/clang

We would need them if the BOOSTIO branch gets integrated into the main branch, but since this may be interesting for everyone anyway, I am raising the issue here first.

I would like to request creation of an external for CondFormats/JetMETObjects/data, as suggested by @ktf and @slava77. This external will contain 2 ROOT files for quark-gluon tagging and 2 for pileup jet ID, as part of the "jet toolbox":

http://home.fnal.gov/~jstupak/files/ReducedHisto_2012.root
http://home.fnal.gov/~jstupak/files/ReducedHisto_2012_CHS.root

http://home.fnal.gov/~jstupak/files/TMVAClassificationCategory_JetID_53X_chs_Dec2012.weights.xml
http://home.fnal.gov/~jstupak/files/TMVAClassificationCategory_JetID_53X_Dec2012.weights.xml

CMS private patch on top of Geant4 10.0p02 has been prepared:

http:://cern.ch/vnivanch/verification/verification/data/ geant4-10.0p02-cms01.patch

This patch include following modifications:

• several fixes of data race (two identified by CMS, other by Geant4 developers);
• fix of integer overflow in pre-compound model
• fix of acces out of array boundary in Binary cascade
• fix of electromagnetic/highenergy library - models describing very rare processes not used by default

These fixes should not affect mainstream production because all are technical. Should be applided both on 7_2_X and 7_2_GEANT10