Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Files for validation pipeline #3

Open
wants to merge 76 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
76 commits
Select commit Hold shift + click to select a range
8f544bf
ADD script for validation plots for standalone ARC detector
Jul 31, 2024
8f8ea4e
ADD structure and new scripts for validation pipeline
Aug 5, 2024
b5f0062
Delete scripts/ARC_val_plots.py
enlupi Aug 5, 2024
0bc87fd
FIX typo in steering file name
Aug 5, 2024
b530dfd
Merge branch 'new_script_validation' of github.com:enlupi/key4hep-rec…
Aug 5, 2024
29a1522
DELETE useless git clone
Aug 6, 2024
b1e908d
FIX errors when using new steering file from CLDConfig
Aug 6, 2024
8216670
ADD comparison for histos in files
Aug 7, 2024
53d94e3
ADD use ROOT files for histograms and comparison
Aug 7, 2024
02f8ee2
FIX k4geo repo
Aug 8, 2024
e7568a2
FIX k4geo repo
Aug 8, 2024
9a20f6b
ADD single script for all stages
Aug 8, 2024
7869ad4
FIX typo with reference directory name
Aug 9, 2024
b868825
FIX compact file for simulation
Aug 9, 2024
1092189
FIX reference file directory
Aug 9, 2024
3b195b3
ADD redirect simulation output to log file
Aug 9, 2024
6bdff56
ADD verbosity
Aug 9, 2024
063d3c6
ADD absence of reference histogram case
Aug 9, 2024
ef1e67a
FIX histogram names
Aug 9, 2024
8ace834
ADD kolmogorov and identical tests
Aug 12, 2024
b759ff0
check if histograms are perfectly identical'
Aug 12, 2024
f86566c
ADD ALLEGRO test
Aug 12, 2024
6940fb7
Change repository structure
Aug 12, 2024
7e0149e
add clarity to print output
Aug 12, 2024
d8ef395
change dirctory structure
Aug 12, 2024
97fd774
FIX working directory
Aug 12, 2024
74a60d6
FIX histogram initialization
Aug 12, 2024
f76af95
FIX sourcing nightlies
Aug 12, 2024
09f4aff
FIX path for compare_histos
Aug 12, 2024
f3fcc2b
FIX plot params keys
Aug 12, 2024
7d44000
ADD info for ECalBarrelModuleThetaMergedPosition
Aug 13, 2024
d27bf0b
Merged www and web folders from new_web_test-branch
Aug 15, 2024
610a05b
Delete web/templates/section_index.html
enlupi Aug 15, 2024
d41481d
Delete www/static/navbar.css
enlupi Aug 15, 2024
a9c32b1
Delete www/static/plot.css
enlupi Aug 15, 2024
08d18a2
Merge branch 'add_comparison' into validation_project
Aug 15, 2024
c73c959
Merge with add_comparison-branch
Aug 15, 2024
c46e779
Refactor analysis scripts
Aug 16, 2024
968cf00
delete wrong files
Aug 16, 2024
79e9173
Delete scripts/FCCee/ALLEGRO/ALLEGRO_o1_v03/ALLEGRO_val_plots.py
enlupi Aug 16, 2024
f0ad06d
Make changes to fit new schema
Aug 16, 2024
80108ea
FIX name of python script to make root file
Aug 19, 2024
526001a
eliminate source stack and fix run_digi_reco script directory
Aug 19, 2024
e669d60
Change how to make histogram list
Aug 19, 2024
ced5c9b
ADD IDEA detector tests
Aug 19, 2024
9d872f2
FIX IDEA script name
Aug 19, 2024
222bc03
FIX variable name
Aug 19, 2024
6d47b34
ADD edge case where file or directory does not exist
Aug 19, 2024
ae856a1
ADD check using KEY4HEP_STACK and take out sed for sim_digi_reco script
Aug 20, 2024
632980f
ADD absolute path to key4hep-reco-validation scripts call
Aug 20, 2024
b10ce32
FIX typo
Aug 21, 2024
0f5610e
change ARC_standalone to adapt to new format
Aug 21, 2024
60800c8
Use CLDCONFIG from stack and not GitHub repo
Aug 21, 2024
37509e2
FIX k4geo path
Aug 21, 2024
6468550
Move plot part inside pipeline
Aug 21, 2024
fa663e6
FIX y axis label
Aug 27, 2024
bdcd756
CHANGE plotting to ROOT functions instead of matplotlib
enlupi Aug 29, 2024
76baedc
FIX typo
enlupi Aug 29, 2024
7197cb5
FIX plot color, legend and stats box
Aug 29, 2024
e1a691d
ADD HIST option when drawing
Aug 29, 2024
999ed62
FIX histogram titles
Aug 29, 2024
a8bb985
Change root TFile creation
Aug 30, 2024
a85a03c
Make lines thicker
Aug 30, 2024
097988d
Change root TFile creation
Aug 30, 2024
05b6278
remove file
Aug 30, 2024
48dbe37
remove file
Aug 30, 2024
accb98a
ADD MuonSystem and Lumical Histos
Sep 4, 2024
b00dca4
Delete scripts/__pycache__ directory
enlupi Sep 4, 2024
fd42e51
Remove unused lines in scripts/FCCee/ALLEGRO/ALLEGRO_o1_v03/ALLEGRO_m…
enlupi Sep 11, 2024
40fdb4c
REMOVE not used intermediate variable
Sep 11, 2024
a4a66a2
Merge branch 'validation_project' of github.com:enlupi/key4hep-reco-v…
Sep 11, 2024
8afd194
Change style of comment blocks
Sep 11, 2024
575bea3
ADD send_mail script
Sep 11, 2024
d6365a2
ADD set -e command for error handling within pipeline (and fix consis…
Sep 11, 2024
b5fd6e1
specify name of pythonn script directly
Sep 11, 2024
7734a53
ADD changes to send emails
Sep 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 129 additions & 0 deletions scripts/FCCee/ALLEGRO/ALLEGRO_o1_v03/ALLEGRO_make_TH1.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
import ROOT
from podio import root_io
import dd4hep as dd4hepModule
from ROOT import dd4hep

import argparse

import os
from dd4hep import Detector
import DDRec


def make_TH1_file(args):

# create output ROOT file
outputFile = ROOT.TFile(args.outputFile, "RECREATE")

# set file reader
podio_reader = root_io.Reader(args.inputFile)

# set list of directories in ROOT file (one per subsystem to check)
dir_list = []
# set list of (list of) histograms created (one list per subsystem to check)
histo_list = []



############# BEGIN: ECal Barrel Histogram Definition #############

dir_ECalBarrel = outputFile.mkdir("ECalBarrel")

hist_ccE = ROOT.TH1F("h_CaloCluster_E",
"CaloCluster Energy;Energy [MeV];Counts / 0.15 MeV",
100, 0, 15)
hist_ctcE = ROOT.TH1F("h_CaloTopoCluster_E",
"CaloTopoCluster Energy;Energy [MeV];Counts / 0.15 MeV",
100, 0, 15)
hist_ecal_totE = ROOT.TH1F("h_ECalBarrelModuleThetaMergedPosition_totE",
"ECalBarrelModuleThetaMergedPosition total Energy per evt;Energy [MeV];Counts / 0.15 MeV",
100, 0, 15)
hist_ecal_posX = ROOT.TH1F("h_ECalBarrelModuleThetaMergedPosition_posX",
"ECalBarrelModuleThetaMergedPosition position X;X [mm];Counts / 37 mm",
150, -2770, 2770)
hist_ecal_posY = ROOT.TH1F("h_ECalBarrelModuleThetaMergedPosition_posY",
"ECalBarrelModuleThetaMergedPosition position Y;Y [mm];Counts / 37 mm",
150, -2770, 2770)
hist_ecal_posZ = ROOT.TH1F("h_ECalBarrelModuleThetaMergedPosition_posZ",
"ECalBarrelModuleThetaMergedPosition position Z;Z [mm];Counts / 41 mm",
150, -3100, 3100)

dir_list.append(dir_ECalBarrel)
histo_list.append([hist_ccE,
hist_ctcE,
hist_ecal_totE,
hist_ecal_posX,
hist_ecal_posY,
hist_ecal_posZ])

############## END: ECal Barrel Histogram Definition ##############




# Loop over dataset
for event in podio_reader.get("events"):

################## BEGIN: ECal Barrel Event Loop ##################

for calo in event.get("CaloClusters"):
hist_ccE.Fill(calo.getEnergy())

for calo in event.get("CaloTopoClusters"):
hist_ctcE.Fill(calo.energy())

energy = 0
for ecal in event.get("ECalBarrelModuleThetaMergedPositioned"):
energy += ecal.energy()
hist_ecal_posX.Fill(ecal.position().x)
hist_ecal_posY.Fill(ecal.position().y)
hist_ecal_posZ.Fill(ecal.position().z)
hist_ecal_totE.Fill(energy)

################## END: ECal Barrel Event Loop ##################



# normalize if desired
if args.norm:
n_evts = len(podio_reader.get("events"))
factor = 1./n_evts
for l in histo_list:
for h in l:
h.Scale(factor)

# save to file
for dir, h_list in zip(dir_list, histo_list):
dir.cd()
for h in h_list:
h.Write()

outputFile.Close()

return






#########################################################################

if __name__ == "__main__":
parser = argparse.ArgumentParser(
description="Process simulation"
)
parser.add_argument('-f', "--inputFile", type=str,
help='The name of the simulation file to be processed', default='ALLEGRO_sim_digi_reco.root')
parser.add_argument('-o', "--outputFile", type=str,
help='The name of the ROOT file where to save output histograms', default='results.root')
parser.add_argument('--norm', action='store_true',
help='Normalize output histograms by number of events')
args = parser.parse_args()

make_TH1_file(args)





26 changes: 26 additions & 0 deletions scripts/FCCee/ALLEGRO/ALLEGRO_o1_v03/ALLEGRO_o1_v03_script.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# enable exit on error to check correct script execution from within pipeline
set -e

# setup phase
echo "SETUP PHASE:"

[ -z "$KEY4HEP_STACK" ] && source /cvmfs/sw-nightlies.hsf.org/key4hep/setup.sh


# simulation phase
echo "SIM-DIGI-RECO PHASE:"

echo "Starting script..."
source $FCCCONFIG/share/FCC-config/FullSim/ALLEGRO/ALLEGRO_o1_v03/ctest_sim_digi_reco.sh


# analyze simulation file
echo "ANALYSIS PHASE:"

echo "Starting analysis script..."
python $WORKAREA/key4hep-reco-validation/scripts/FCCee/ALLEGRO/ALLEGRO_o1_v03/ALLEGRO_make_TH1.py \
-f ALLEGRO_sim_digi_reco.root -o results.root
echo "Script executed successfully"



149 changes: 149 additions & 0 deletions scripts/FCCee/CLD/CLD_o3_v01/ARC_make_TH1.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
import ROOT
from podio import root_io
import dd4hep as dd4hepModule
from ROOT import dd4hep
import argparse
import numpy as np
import os
from dd4hep import Detector
import DDRec

def find_directory_upwards(start_dir, target_dir_name):
"""
Recursively searches upwards from the start directory for a directory with the specified name.

:param start_dir: The starting directory path.
:param target_dir_name: The name of the directory to search for.
:return: The path to the found directory or None if not found.
"""
current_dir = os.path.abspath(start_dir)

while True:
# Check if the target directory exists in the current directory
if target_dir_name in os.listdir(current_dir):
possible_match = os.path.join(current_dir, target_dir_name)
if os.path.isdir(possible_match):
return possible_match

# Move up one level in the directory tree
parent_dir = os.path.dirname(current_dir)

# If we have reached the root directory and haven't found the directory, return None
if current_dir == parent_dir:
return None

current_dir = parent_dir


def make_TH1_file(args):

# create output ROOT file
outputFile = ROOT.TFile(args.outputFile, "UPDATE")

# set file reader
podio_reader = root_io.Reader(args.inputFile)

# get detector description for cell id decoding
k4geo = find_directory_upwards('./', 'k4geo')
theDetector = Detector.getInstance()
theDetector.fromXML(k4geo+"/test/compact/ARC_standalone_o1_v01.xml")
idposConv = DDRec.CellIDPositionConverter(theDetector)

# set list of directories in ROOT file (one per subsystem to check)
dir_list = []
# set list of (list of) histograms created (one list per subsystem to check)
histo_list = []



########### BEGIN: ARC standalone Histogram Definition ############

dir_ARC = outputFile.mkdir("ARC_standalone")

hist_ARC_nPh = ROOT.TH1F("h_ARC_nPh",
"Total number of photon counts per event;Number of Photons;Photon count / 5",
50, 0, 250)
hist_ARC_theta = ROOT.TH1F("h_ARC_theta",
"Photons counts vs. #theta;Polar angle #theta;Photon count / 35 mrad",
90, 0, np.pi)
hist_ARC_1stHit = ROOT.TH1F("h_ARC_1stHit",
"Photons counts vs. #theta of incoming particle;Polar angle #theta;Photon count / 35 mrad",
90, 0, np.pi)

dir_list.append(dir_ARC)
histo_list.append([hist_ARC_nPh,
hist_ARC_theta,
hist_ARC_1stHit])

############ END: ARC standalone Histogram Definition #############




# loop over dataset
for event in podio_reader.get("events"):

############### BEGIN: ARC standalone Event Loop ################

n_ph = 0

p = (event.get("MCParticles"))[0]
mom = ROOT.TVector3(p.getMomentum().x, p.getMomentum().y, p.getMomentum().z)
theta_1stHit = mom.Theta()

for arc_hit in event.get("ArcCollection"):
particle = arc_hit.getMCParticle()
pdgID = particle.getPDG()
if pdgID == 22 or pdgID == -22:
n_ph += 1
cellID = arc_hit.getCellID()
x = idposConv.position(cellID)
theta = x.theta()

hist_ARC_theta.Fill(theta)

hist_ARC_nPh.Fill(n_ph)
hist_ARC_1stHit.Fill(theta_1stHit, n_ph)

################ END: ARC standalone Event Loop #################


if args.norm:
n_evts = len(podio_reader.get("events"))
factor = 1./n_evts
for l in histo_list:
for h in l:
h.Scale(factor)

# save to file
for dir, h_list in zip(dir_list, histo_list):
dir.cd()
for h in h_list:
h.Write()

outputFile.Close()

return


#########################################################################

if __name__ == "__main__":
parser = argparse.ArgumentParser(
description="Process simulation"
)
parser.add_argument('-f', "--inputFile", type=str,
help='The name of the simulation file to be processed', default='ARC_sim.root')
parser.add_argument('-o', "--outputFile", type=str,
help='The name of the ROOT file where to save output histograms', default='results.root')
parser.add_argument('--norm', action='store_true',
help='Normalize output histograms by number of events')
args = parser.parse_args()

ph_presence = make_TH1_file(args)
print(ph_presence)





42 changes: 42 additions & 0 deletions scripts/FCCee/CLD/CLD_o3_v01/CLD_o3_v01_script.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# enable exit on error to check correct script execution from within pipeline
set -e

# setup phase
echo "SETUP PHASE:"

[ -z "$KEY4HEP_STACK" ] && source /cvmfs/sw-nightlies.hsf.org/key4hep/setup.sh

echo "Downloading necessary Github repos..."
git clone https://github.com/key4hep/k4geo.git
K4GEO_PATH=$(realpath k4geo)
echo "Download terminated - setup stage successful"


# simulation phase
echo "SIMULATION PHASE:"

cd $CLDCONFIG/share/CLDConfig
echo "Starting simulation..."
ddsim --steeringFile cld_arc_steer.py \
--compactFile $K4GEO_PATH/test/compact/ARC_standalone_o1_v01.xml \
--enableGun --gun.distribution "cos(theta)" \
--gun.energy "20*GeV" --gun.particle proton \
--numberOfEvents $NUMBER_OF_EVENTS \
--outputFile $WORKAREA/$GEOMETRY/$VERSION/ARC_sim.root \
--random.enableEventSeed --random.seed 42 \
--part.userParticleHandler='' > $WORKAREA/$GEOMETRY/$VERSION/ddsim_log.txt
echo "Simulation ended successfully"
cd $WORKAREA/$GEOMETRY/$VERSION


# analyze simulation file
echo "ANALYSIS PHASE:"

echo "Starting analysis script..."
python $WORKAREA/key4hep-reco-validation/scripts/FCCee/CLD/CLD_o3_v01/ARC_make_TH1.py \
-f ARC_sim.root -o results.root
echo "Script executed successfully"




Loading