-
Notifications
You must be signed in to change notification settings - Fork 11
ProdV3
Konstantin Androsov edited this page Jul 30, 2017
·
11 revisions
- Install framework on lxplus in a prod workspace directory without creating CMSSW release area
curl -s https://raw.githubusercontent.com/hh-italian-group/hh-bbtautau/master/Run/install_framework.sh | bash -s prod- Check framework production functionality interactively for a few samples
cd CMSSW_8_0_28/src
cmsenv
# Radion 250 sample
echo /store/mc/RunIISummer16MiniAODv2/GluGluToRadionToHHTo2B2Tau_M-250_narrow_13TeV-madgraph/MINIAODSIM/PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/110000/A06FE4CA-27DB-E611-9246-549F3525C380.root > Radion_250.txt
cmsRun h-tautau/Production/python/Production.py fileList=Radion_250.txt applyTriggerMatch=True sampleType=Summer16MC ReRunJEC=True globalTag=80X_mcRun2_asymptotic_2016_TrancheIV_v6 tupleOutput=eventTuple_Radion_250.root maxEvents=1000
# Single electron Run2016B
echo /store/data/Run2016B/SingleElectron/MINIAOD/03Feb2017_ver2-v2/110000/003B2C1F-50EB-E611-A8F1-002590E2D9FE.root > SingleElectron_B.txt
cmsRun h-tautau/Production/python/Production.py fileList=SingleElectron_B.txt anaChannels=eTau applyTriggerMatch=True sampleType=Run2016 ReRunJEC=True globalTag=80X_dataRun2_2016SeptRepro_v7 tupleOutput=eventTuple_SingleElectronB.root saveGenTopInfo=False saveGenBosonInfo=False saveGenJetInfo=False energyScales=Central lumiFile=h-tautau/Production/json/Cert_271036-284044_13TeV_PromptReco_Collisions16_JSON.txt maxEvents=1000- Install framework on the stage out site (e.g. Pisa)
curl -s https://raw.githubusercontent.com/hh-italian-group/hh-bbtautau/master/Run/install_framework.sh | bash -s prodEach time after login:
source /cvmfs/cms.cern.ch/crab3/crab.sh
voms-proxy-init --voms cms --valid 168:00
cd CMSSW_DIR/src
cmsenv
cd h-tautau/Production/crab- Submit jobs
./submit.py --work-area work-area --cfg ../python/Production.py --site T2_IT_Pisa --output hh_bbtautau_prod_v3 config1 [config2] ...- Periodically (1-2 times per day) check jobs status
./multicrab.py --workArea work-area --crabCmd statusAnalyze output of the status command for each task:
1. If few jobs were failed without any persistent pattern, resubmit them:
```shell
crab resubmit -d work-area/TASK_AREA
```
2. If significant amount of jobs are failing, one should investigate the reason and take actions accordingly. For more details see the CRAB troubleshoot section below.
3. If all jobs are successfully finished, move task area from 'work-area' into 'finished' directory (create it if needed).
```shell
# mkdir -p finished
mv work-area/TASK_AREA finished/
```
4. If there is no reasonable hope that some jobs will be successfully finished, move task area from 'work-area' into 'finished-partial' directory (create it if needed).
Before moving the directory, make sure that all jobs has or 'failed' or 'finished', otherwise wait (use kill command, if necessary).
Create recovery task for the failed jobs (see CRAB troubleshoot section).
```shell
# mkdir -p finished-partial
mv work-area/TASK_AREA finished-partial/
```
5. Create job lists for jobs in 'finished' and 'finished-partial' directories and transfer them into stage out server
```shell
if -d current-check ; then mv current-check prev-check ; fi
mkdir current-check
for NAME in finished* ; do ./create_job_list.sh $NAME | sort > current-check/$NAME.txt ; done
```
6. Update prod_v3 spreadsheet accordingly. 'finished-partial' task should be considered as not complete, so no updates in the spreadsheet are needed.
- TODO Submit merge jobs output files on the stage out server.
- For partially finished tasks wait for the recovery task to be finished before start merging.
Use find_new_jobs.sh, submit_tuple_hadd.sh, collect_tuple_hadd.sh 4. TODO Split large merged files into N parts in order to satisfy cernbox requirement that file size should be less than 8GB. use submit_job.sh split_tree.py
./h-tautau/Instruments/submit_tuple_hadd.sh production current-check/finished.txt output/07_30_finished /gpfs/ddn/srm/cms/store/user/kandroso/hh_bbtautau_prod_v3- Transfer tuple files into the central prod_v3 cernbox directory. Use sub-directory Full. Update prod_v3 spreadsheet accordingly.
TODO