[mg] gridpack production on condor, T2 UCSD about genproductions HOT 11 CLOSED

@danbarto I suspect you'll need to involve your T2 support. @khurtado may also have suggestions.

If you want to make the issue easier to debug, you can uncomment out line 1105 of bin/internal/cluster.py

open('submit_condor','w').write(text % dico)

so that the condor submit file is actually stored. I think it gets stored in the SubProcesses folders, but you might also want to make it an explicit path. You can then run

condor_submit submit_condor

on this file, which should fail in the same way, but will at least be more useful for the T2 support to help debug. One thing you might try is adding the line

GetEnv 1

to the submit file, as my first impression is that there could be an environment issue.

By the way, can you confirm that the older scripts that you were using before at UCSD are still working? I don't understand why this issue would show up now but not previously.

from genproductions.

Thanks for the reply. I did manually run the condor job as you suggested, and the logs are here:
http://cmsdoc.cern.ch/~dspitzba/logs/condor.out
http://cmsdoc.cern.ch/~dspitzba/logs/condor.err
http://cmsdoc.cern.ch/~dspitzba/logs/condor.log
Condor.out shows the same error about loading the shared gfortran library.
getenv=True was already set in the condor submit file (http://cmsdoc.cern.ch/~dspitzba/logs/submit_condor in case you are interested).

About the older scripts I have to point to @vdutta since I only have started using the UCSD T2.

from genproductions.

Does it work for you locally?

from genproductions.

@kdlong regarding the older scripts, they do still work but with an older version of genproductions and with a patch we used to apply specific to UCSD:
https://github.com/CMS-SUS-XPAG/GenLHEfiles/blob/master/GridpackWorkflow/test/scripts/ucsd.patch

from genproductions.

Yes, the gridpack creation runs locally without any problems

from genproductions.

@vdutta @danbarto, can you modify the old patch to be applied on top of the current branch and see if that fixes your issue? Things have been moved around a decent bit, but most of the changes that were made apply to the job management, so adjusting the submit file to be more UCSD specific should be no problem at all.

By the way, we have no problem at all with this type of customization and can even make it more centrally supported if that helps. For now, you should be able to just add the (updated) patch into the patches folder, just make sure it doesn't conflict with anything else. What we don't want is a fully independent repository that can conflict with the central one.

from genproductions.

Coming back to this: I think the work being done by @AndreasAlbert @pmandrik could be extended to cover such instances of customization. For example, we could add an option condor config customization file that gets read and appended to the submit config used internal in MadGraph. I would be interested in being able to add simple options to the submit for Wisconsin. How about at UCSD, would you be willing to help look into this?

from genproductions.

I'm referencing the cluster module changes made in #1662 by the way.

from genproductions.

We are currently working on resolving the gfortran issue at UCSD. As soon as this is done I can have a look, however, I'm not a condor expert so it might take some time.

from genproductions.

I finally resolved the issue by creating an additional patch:
https://github.com/danbarto/genproductions/blob/master/bin/MadGraph5_aMCatNLO/patches/0021-fix-libraries-T2-UCSD.patch
There's in fact no need to change the condor configuration for UCSD, but there's just a problem with LD_LIBRARY_PATH not being set.
Should I open a PR with this patch, or should it be modified?

from genproductions.

Hi, sorry I overlooked this. I think it would be best to keep such patches custom. However, note that we have added some support for custom patches starting in 2.6.0.

https://github.com/cms-sw/genproductions/blob/mg26x/bin/MadGraph5_aMCatNLO/gridpack_generation.sh#L123-L126

You could add this file to your cards directory and upload it to genproductions when you add your usual card, then it will be clear that the samples were produced with this modification, which is good for bookkeeping.

from genproductions.