PYTHON framework for the analysis of ROOT TTree data using uproot for the IO and awkward-array for the columnar data analysis.
The tool is developed for the analysis of FastPUPPI but should work with any kind of flat ntuples.
The tool can be run on any private machines using just python, ROOT, pip and venv.
If you plan to run it on lxplus you might want to look at the point 1 below.
You can create a venv with different procedures, using explicitly venv (see paragraph 2 below) or using virtualenvwrapper (see paragraph 3).
This step is lxplus specific, givin access to a more recent python (>= 3.10) and ROOT version.
Edit/skip it accordingly for your specific system.
The current configuration has been tested only on el7 and el8.
source setup_lxplus.sh
python3.10 -m venv <venvname>
source <venvname>/bin/activate
pip install -r requirements.txtThis stetp needs to be done only once for your account and can be done with whatever python version is in use in the system.
For some reason the current CMSSW scrips seems to deliver an inconsistent setup of virtualenv and virtualenvwrapper, for this reason we force a new installation in ~/.local using:
pip install --ignore-installed --user virtualenv==15.1.0 virtualenvwrapper
For a more complete overview of the procedure you can refer to
virtualenvwrapper installation instructions
For starting using virtualenvwrapper
source setVirtualEnvWrapper.sh
The first time you will have to create the actual instance of the virtualenv:
mkvirtualenv --system-site-packages -p `which python3.10` -r requirements.txt <venvname>
This step is lxplus specific, givin access to a more recent python and root version.
Edit/skip it accordingly for your specific system.
source setup_lxplus.sh
Depending on how you created the virtualenv you need to activate it using one of the folloiwing:
source <venvname>/bin/activateFor starting using virtualenvwrapper
`source setVirtualEnvWrapper.sh`
`workon <venvname>`(lsvirtualenv is your friend in case you forgot the name).
The main script is analyzeNtuples.py:
python analyzeNtuples.py --help
An example of how to run it:
python analyzeNtuples.py -f cfg/egvalid.yaml -i cfg/datasets/ntpfp_131Xv3.yaml -p egmenu -s doubleele_flat1to100_PU200 -n 1000 -d 0
The analysis is defined by a yaml file and a python module of the same name. They define a number of collection of plotters which read some data and fill a set of plots for a list of data selections. In case gen matching is needed the same plots are filled for all the combinations of data and gen selections specified in the configuration.
One of the collection is specified via command line arguments (-p option).
Data are represented by collections of objects which can be read from ROOT::TTree files or filled on the fly. They are processed by plotters which creates set of histograms for different selections of the data collections.
The plotters, the histograms and the various selections are defined in the configuration python file.
The ouput histograms are saved in the output file following a naming convention:
<Histo class name>/<collection name>_<selection name>_<histo name>
In case gen matching is performed the naming convention becomes:
<Histo class name>/<collection name>_<selection name>_<gen collection name>_<gen selection name>_<histo name>
The histogram classes handle saving and reading histograms to/from file transparently.
The configuration is handled by 2 yaml files.
One specifying
- output directories
- versioning of the plots
- collections of samples, i.e. group of samples to be processed homogeneously: for each collection the list of plotters (see below) to be run is provided.
The other prividing
- details of the input samples (location of the ntuple files)
Example of configuration file can be found in:
So you can run the same set of plotters on different input ntuples.
The list of branches to be read and converted to Awkward Arrays format is specified in the module
Instantiating objects of class DFCollection. What is actually read event by event depends anyhow on which plotters are actually instantiated (collections are read on-demand).
For each collection, a function adding columns beyond those in the root file can be defined.
New collections can be created for example combining those read from the root file.
Selections are defined as strings in the module:
Different collections are defined for different objects and/or different purposes. The selections have a name which is used for the histogram naming (see below). Selections are used by the plotters.
Selections can be combined and retrieved via regular expressions in the configuration of the plotters.
The actual functionality of accessing the objects, filtering them according to the selections and filling histograms is provided by the plotter classes. The base ones are defined in the module:
Basic plotters are already available, most likely you just need to instantiate one of them (or a collection of them) using the DFCollection instance you are interested in.
Which collection is run for which sample is steered by the configuration file.
The plotters access one or more collections, select them in several different ways, book and fill the histograms (see below).
Histograms are handled in the module:
There are different classes of histograms depending on the input object and on the purpose.
To add a new histogram to an existing class it is enough to add it in the corresponding constructor and in the fill module. The writing of the histos to files is handled transparently.
The histogram naming follows the convention:
<ObjectName>_<SelectionName>_<GenSelectionName>_<HistoName>
This is assumed in all the plotters and in the code to actually draw the histograms.
Histograms are coputed using boost histograms via the hist but saved as ROOT histograms.
Of course you can use your favorite set of tools: histograms are coputed using boost histograms via the hist but saved as ROOT histograms.
The draw.py script allows to call different drawing modules defined for the various analysis modules. Given a list of files the will retrieve histograms by collection and by selection name and plot them. The resulting png and pdf files are saved to the target directory specified via command line.
Additionally, interactive plotting can be done using the jupyter notebooks in plot-drawing-tools.
cd ntuple-analysis
git clone [email protected]:cerminar/plot-drawing-tools.git
jupyter-notebook
- Running GEN matching to compute efficiency on e/g menu objects and draw plots:
python analyzeNtuples.py -f cfg/eg_genmatch.yaml -i cfg/datasets/ntpfp_131Xv3.yaml -p egmenu -s doubleele_flat1to100_PU200 -n 1000 -d 0
python draw.py -m cfg/eg_genmatch_draw.py -w egmenu_ele --input-files path/file1.root:label1,path/file2.root:label2 --target-dir /Users/cerminar/CERNbox/www/plots/test2/- Runnig rate computations on e/g menu objects:
python analyzeNtuples.py -f cfg/eg_rate.yaml -i cfg/datasets/ntpfp_131Xv3.yaml -p rate_menu -s nugun_alleta_pu200 -n 1000 -d 0
python draw.py -m cfg/eg_rate_draw.py -w menu_rate --input-files plots/histos_nugun_alleta_pu200_ratemenu_v160A.v131Xv1A.root:menu-v31,plots/histos_nugun_alleta_pu200_ratemenu_v160A.131Xv3.root:menu-v33 --target-dir /Users/cerminar/CERNbox/www/plots/test2/- Running rate computation for main L1T Menu seeds:
python analyzeNtuples.py -f cfg/eg_rate.yaml -i cfg/datasets/ntpfp_131Xv3.yaml -p rate_counter_menu -s nugun_alleta_pu200 -n -1 -d 0
python draw.py -m cfg/eg_rate_draw.py -w menu_ratecounter --input-files plots/histos_nugun_alleta_pu200_ratemenu_v160A.131Xv3.root:menu,plots/histos_nugun_alleta_pu200_egratemenu_v160A.131Xv3M.root:menu-emu --target-dir /Users/cerminar/CERNbox/www/plots/fp131Xv3-CMSSW14.0.X-GCTEmu- Running GEN matching to compute efficiency on HGC TPs objects:
python analyzeNtuples.py -f cfg/hgctps.yaml -i cfg/datasets/ntpfp_v100.yaml -p genmatch -s doubleele_flat1to100_PU200 -n 1000 -d 0- Running rate computations on HGC clusters
python analyzeNtuples.py -f cfg/hgctps.yaml -i cfg/datasets/ntpfp_v100.yaml -p rate -s doubleele_flat1to100_PU200 -n 1000 -d 0- Computing HGC cluster occupancies per CTL1 region
python analyzeNtuples.py -f cfg/l1ct_occupancy.yaml -i cfg/datasets/ntpfp_v100.yaml -p tps -s doubleele_flat1to100_PU200 -n 1000 -d 0- Producing a further
ROOTntuple with HGC clusters and matched GEN information (e.g. for model training)
python analyzeNtuples.py -f cfg/hgcIdTuples.yaml -i cfg/datasets/ntpfp_131Xv3.yaml -p egid -s ttbar_PU200 -n 1000 -d 0- Computing efficiency on jets
python analyzeNtuples.py -f cfg/jets.yaml -i cfg/datasets/ntpfp_131Xv3.yaml -p ak4_jets -s ttbar_PU200 -n -1 -d 0
python draw.py -m cfg/jets_draw.py --target-dir /Users/cerminar/CERNbox/www/plots/tj -w eff --input-files /Users/cerminar/cernbox/hgcal/CMSSW1015/plots/histos_ttbar_PU200_jets_v200A.131Xv3i.root:testI can't figure out how to do some manipulation using the awkward array or uproot....you can take a look at examples and play witht the arrays in:
plot-drawing-tools/blob/master/eventloop-uproot-ak.ipynb