HEPHero - Framework for the DESY-CBPF-UERJ collaboration

Initial setup

Set up your environment: (only once in your life)
CERN - add the lines below to the file /afs/cern.ch/user/${USER:0:1}/${USER}/.bashrc and restart your session.
DESY - add the lines below to the file /afs/desy.de/user/${USER:0:1}/${USER}/.zshrc and restart your session. (Create the file if it doesn't exist)
UERJ - add the lines below to the file /home/${USER}/.bashrc and restart your session.
PC (Personal Computer) - add the lines below to the file /home/${USER}/.bashrc and restart your session.

export HEP_OUTPATH=<place the full path to a directory that will store the outputs>
export REDIRECTOR=<place the redirector suitable to your geographic region>
export MACHINES=<place the organization name, owner of the computing resources>

alias hepenv='source /afs/cern.ch/work/g/gcorreia/public/hepenv_setup.sh'  #(CERN)
alias hepenv='source /afs/desy.de/user/g/gcorreia/public/hepenv_setup.sh'  #(DESY)
alias hepenv='source /mnt/hadoop/cms/store/user/gcorreia/hepenv_setup.sh'  #(UERJ)
alias hepenv='source $HEP_OUTPATH/hepenv_setup.sh'                         #(PC)
alias cernenv='source $HEP_OUTPATH/container_setup.sh'                     #(PC)

Possible outpaths:

• At CERN, use a folder inside your eos area
• At DESY, use a folder inside your dust area
• At UERJ, you must create and define your outpath as: /home/username/output
• At PC, use any folder inside your home area

Possible redirectors (only used at CERN):

• cmsxrootd.fnal.gov (USA)
• xrootd-cms.infn.it (Europe/Asia)
• cms-xrd-global.cern.ch (Global)

Possible machines:

• CERN
• DESY
• UERJ

Examples

export HEP_OUTPATH=/eos/user/g/gcorea/output
export REDIRECTOR=xrootd-cms.infn.it
export MACHINES=CERN

export HEP_OUTPATH=/nfs/dust/cms/user/gcorea/output
export REDIRECTOR=None
export MACHINES=DESY

export HEP_OUTPATH=/home/gcorea/output
export REDIRECTOR=None
export MACHINES=UERJ

export HEP_OUTPATH=/home/gcorea/output
export REDIRECTOR=None
export MACHINES=PC

Quick start

Inside your private or home area (NOT in the eos or dust area and NOT inside a CMSSW release), download the code.

git clone [email protected]:DESY-CBPF-UERJ/HEPHero.git

Source the hepenv environment before work with the HEPHero:

hepenv

Set up the runSelection.py to one of the available analysis folders inside the HEPHero directory:

python setAnalysis.py -a bbZDM_Lep_R2

Enter in the HEPHero directory and compile the code (running cmake is necessary only at the first time):

cd HEPHero
cmake .
make -j 8

Create a template (if it doesn't exist) for a new anafile called Test and integrate it to the framework:

./addSelection.sh Test

Dissociate Test from the framework (the anafile is not deleted):

./rmSelection.sh Test

You can check for different cases [m= 0(signal), 1-4(bkg all years), 5(data)] if your anafile is working as intended using the test datasets:

python runSelection.py -c m

Know how many jobs the code is setted to process:

python runSelection.py -j -1

Produce a list of all jobs the code is setted to process:

python runSelection.py -j -2

Produce a list of all missing datsets by the time of the text files creation:

python runSelection.py -j -3

(Only at DESY) Produce a list of all files that exist but are missing at DESY:

python runSelection.py -j -4

Run the job in the nth position of the list:

python runSelection.py -j n

Submiting condor jobs

If you have permission to deploy condor jobs, you can run your code in each dataset as a job.

1. See all flavours available for the jobs
2. Submit all the N jobs the code is setted to process (need to provide the proxy)

./submit_jobs.sh help
./submit_jobs.sh flavour N

Checking and processing condor jobs results

First, go to tools directory.

cd tools

Check integrity of jobs of the selection Test and period 0_16:

python checker.py -s Test -p 0_16

Check a specific dataset:

python checker.py -s Test -p 0_16 -d TTTo2L2Nu

If you want to remove bad jobs, type:

python remove_jobs.py -s Test -l <list of bad jobs>

Once all jobs are good, you can group them by typing:

python grouper.py -s Test -p 0_16

Example for all periods of Run 2:

python checker.py -s Test -p 0_16
python checker.py -s Test -p 1_16
python checker.py -s Test -p 0_17
python checker.py -s Test -p 0_18

If your anafile was set to produce systematic histograms, you need to add the syst flag to check and group as well the json files where are stored the histograms. Examples:

python checker.py -s Test -p 0_16 --syst
python grouper.py -s Test -p 0_16 --syst

By default, checker and grouper use the number of CPUs available minus 2. You can force a specific number. For example, using 5 CPUs:

python checker.py -s Test -p 0_16 -c 5
python grouper.py -s Test -p 0_16 -c 5

If the code is crashing, the debug flag can help you to identify the problematic folder:

python checker.py -s Test -p 0_16 --debug
python grouper.py -s Test -p 0_16 --debug

In the checker, the problematic dataset is known. In order to save time, it is recommended to use the debug flag in combination with the name of the dataset you want to investigate.

python checker.py -s Test -p 0_16 -d TTTo2L2Nu --debug

Resubmiting condor jobs

If there are bad jobs in the output directory, they will be written in the tools/resubmit_X.txt file, where X is the period associated with the job. If you desire to resubmit the bad jobs listed in these files, you can use the flag ""--resubmit"" as in the commands below.

Know how many jobs the code is setted to process in the resubmission:

python runSelection.py -j -1 --resubmit

Produce a list of all jobs the code is setted to process in the resubmission:

python runSelection.py -j -2 --resubmit

Resubmit your jobs:

./submit_jobs.sh flavour N --resubmit