Job submission in 0.1.0 (#118)

* renamed simu_config_version * renamed simu_config_version * renamed simu_config_version * renamed simu_config_version * package * package * default * rename * renamed context functions * debug * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * doc * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed * fixed typo * debug * debug * debug * fix --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
XENONnT · Apr 22, 2024 · 7c0374c · 7c0374c
1 parent 8811992
commit 7c0374c
Show file tree

Hide file tree

Showing 8 changed files with 80 additions and 52 deletions.
diff --git a/README.md b/README.md
@@ -1,5 +1,5 @@
 # saltax
-Salting `wfsim` or `fuse` into `strax` data, followed by comparison and analysis. See this [notes](https://xe1t-wiki.lngs.infn.it/doku.php?id=lanqing:ambience_interference_and_sprinkling#raw_records_simu) to see how it serves physics.
+Salting `wfsim` or `fuse` into `strax` data, followed by comparison and analysis. Please check this [notes](https://xe1t-wiki.lngs.infn.it/doku.php?id=lanqing:ambience_interference_and_sprinkling#raw_records_simu) to see how it serves physics.
 
 ## Installation
 ```

diff --git a/jobs/README.md b/jobs/README.md
@@ -2,11 +2,24 @@
 ## Scope
 Job submission scripts.
 ## Structure
-`config.ini`: Configuraiton that you want to change everytime before submitting.
-`submit.py`: wrapper around `utilix.batchq` to submit jobs.
-`job.py`: processing script, just a wrapper around `st.make`.
+- `config.ini`: Configuraiton that you want to change everytime before submitting.
+- `submit.py`: wrapper around `utilix.batchq` to submit jobs.
+- `job.py`: processing script, just a wrapper around `st.make`.
 ## Usage
 Update `config.ini` and run this in a container
 ```
 python submit.py
 ```
+## Tips
+- Unless you are working on AmBe, put `mem_per_cpu = 45000` (MB) should be enough. Otherwise please do `55000`.
+- In `package` you can put either `wfsim` or `fuse`. They are both supported but of course `fuse` is preferred. Please make sure you have the master branch `wfsim` to avoid any photon timing bug.
+- When deciding `generator_name`, take a look in `saltax/saltax/instructions/README.md`.
+- In `recoil`, put NESTID. (`8` is `beta`, `7` is `gamma`, `0` is `WIMP`).
+- Make sure you put `output_folder` to be your own scratch. We will have very huge output because of `records` and `raw_records_simu`.
+- `rate` is in unit of Hz. Be cautious to put anything above 300Hz, since you will start to have time gap between sprinkled event less than the Full Drift Time.
+- `simu_mode = all` means you want to simulate both S1 and S2. Otherwise you specify `s1` or `s2`.
+- Do NOT put space when specifying `runids`. Also don't put `0` in front of numbers!
+- I never tested `cpus_per_task != 1`. Please be cautious if you specify another number.
+- Make sure you change the `username`, `storage_to_patch` and `output_folder` etc before using.
+- In `saltax_mode`, you can put `salt`, `simu` and `data`. Remember that all of them will be computed from the same `records`, so usually you don't want to change `saltax_mode = salt`.
+- When running on `dali`, make sure you put `log_dir` in somewhere under `/dali`. (Otherwise you will ruin the fragile NFS connection!)
diff --git a/jobs/config.ini b/jobs/config.ini
@@ -3,15 +3,16 @@ max_num_submit = 200
 t_sleep = 1
 
 [job]
-container = xenonnt-development.simg
-runids = 48693,48699,48692,47698,49028,49079,49077,49432,49080,49433
+container = xenonnt-montecarlo-development.simg
+runids = 48692,48693,48699,49028,49077,49079,49080,49432,49433
 output_folder = /scratch/midway2/yuanlq/salt/rn220
+package = fuse
 saltax_mode = salt
-faxconf_version = sr0_v4
+simu_config_version = sr1_dev
 generator_name = flat
 recoil = 8
 simu_mode = all
-rate = 50
+rate = 100
 process_data = False
 process_simu = True
 skip_records = False

diff --git a/jobs/job.py b/jobs/job.py
@@ -18,7 +18,8 @@
 config.read("config.ini")
 output_folder = str(config.get("job", "output_folder"))
 saltax_mode = config.get("job", "saltax_mode")
-faxconf_version = config.get("job", "faxconf_version")
+package = config.get("job", "package")
+simu_config_version = config.get("job", "simu_config_version")
 generator_name = config.get("job", "generator_name")
 recoil = config.getint("job", "recoil")
 simu_mode = config.get("job", "simu_mode")
@@ -33,6 +34,14 @@
 storage_to_patch = config.get("job", "storage_to_patch").split(",")
 delete_records = config.getboolean("job", "delete_records")
 
+# Determine context for processing
+if package == "wfsim":
+    context_function = saltax.contexts.sxenonnt
+elif package == "fuse":
+    context_function = saltax.contexts.fxenonnt
+else:
+    raise ValueError("Invalid package name %s" % package)
+
 # Determine whether to process events type plugins or just peak types
 to_process_dtypes_ev = [
     "peaklets",
@@ -82,21 +91,21 @@
 print("Finished importing and config loading, now start to load context.")
 print("Now starting %s context for run %d" % (saltax_mode, runid))
 if rate is None:
-    st = saltax.contexts.sxenonnt(
+    st = context_function(
         runid=runid,
         saltax_mode=saltax_mode,
         output_folder=output_folder,
-        faxconf_version=faxconf_version,
+        simu_config_version=simu_config_version,
         generator_name=generator_name,
         recoil=recoil,
         simu_mode=simu_mode,
     )
 else:
-    st = saltax.contexts.sxenonnt(
+    st = context_function(
         runid=runid,
         saltax_mode=saltax_mode,
         output_folder=output_folder,
-        faxconf_version=faxconf_version,
+        simu_config_version=simu_config_version,
         generator_name=generator_name,
         recoil=recoil,
         simu_mode=simu_mode,
@@ -106,7 +115,14 @@
     for d in storage_to_patch:
         st.storage.append(strax.DataDirectory(d, readonly=True))
 
+if package == "fuse":
+    print("Making microphysics_summary.")
+    st.make(strrunid, "microphysics_summary", progress_bar=True)
+    print("Done with microphysics_summary.")
+    gc.collect()
+print("Making raw_records.")
 st.make(strrunid, "raw_records_simu", progress_bar=True)
+print("Done with raw_records.")
 gc.collect()
 for dt in to_process_dtypes:
     print("Making %s. " % dt)
@@ -135,21 +151,21 @@
         )
         print("Now starting data-only context for run %d" % (runid))
         if rate is None:
-            st = saltax.contexts.sxenonnt(
+            st = context_function(
                 runid=runid,
                 saltax_mode="data",
                 output_folder=output_folder,
-                faxconf_version=faxconf_version,
+                simu_config_version=simu_config_version,
                 generator_name=generator_name,
                 recoil=recoil,
                 simu_mode=simu_mode,
             )
         else:
-            st = saltax.contexts.sxenonnt(
+            st = context_function(
                 runid=runid,
                 saltax_mode="data",
                 output_folder=output_folder,
-                faxconf_version=faxconf_version,
+                simu_config_version=simu_config_version,
                 generator_name=generator_name,
                 recoil=recoil,
                 simu_mode=simu_mode,
@@ -184,21 +200,21 @@
         print("====================")
         print("Now starting simu-only context for run %d" % (runid))
         if rate is None:
-            st = saltax.contexts.sxenonnt(
+            st = context_function(
                 runid=runid,
                 saltax_mode="simu",
                 output_folder=output_folder,
-                faxconf_version=faxconf_version,
+                simu_config_version=simu_config_version,
                 generator_name=generator_name,
                 recoil=recoil,
                 simu_mode=simu_mode,
             )
         else:
-            st = saltax.contexts.sxenonnt(
+            st = context_function(
                 runid=runid,
                 saltax_mode="simu",
                 output_folder=output_folder,
-                faxconf_version=faxconf_version,
+                simu_config_version=simu_config_version,
                 generator_name=generator_name,
                 recoil=recoil,
                 simu_mode=simu_mode,

diff --git a/saltax/README.md b/saltax/README.md
@@ -5,4 +5,4 @@ Core functions of `saltax`.
 - `instructions`: `wfsim` or `fuse` CSV instruction related, including event generators.
 - `match`: tools to match salted truth and reconstucted signals.
 - `plugins`: `straxen`-like plugins for salting, equivalently from `raw_records` to `peaklets`.
-- `contexts.py`: `strax` context for `saltax`. You will want to use `sxenonnt` as default context.
+- `contexts.py`: `strax` context for `saltax`. You will want to use `sxenonnt` (`wfsim` based simulation) or `fxenonnt` (`fuse` based simulation) as default context.
diff --git a/saltax/contexts.py b/saltax/contexts.py
@@ -128,7 +128,7 @@ def xenonnt_salted_fuse(
     output_folder="./fuse_data",
     cut_list=cutax.BasicCuts,
     corrections_version=DEFAULT_XEDOCS_VERSION,
-    simulation_config_file="fuse_config_nt_sr1_dev.json",
+    simu_config_version="sr1_dev",
     run_id_specific_config={
         "gain_model_mc": "gain_model",
         "electron_lifetime_liquid": "elife",
@@ -152,8 +152,8 @@ def xenonnt_salted_fuse(
     :param corrections_version: XENONnT documentation version to use,
         defaults to DEFAULT_XEDOCS_VERSION
     :param cut_list: Cut list to use, defaults to cutax.BasicCuts
-    :param simulation_config_file: File containing simulation
-        configuration
+    :param simu_config_version: simulation configuration version to use,
+        defaults to "sr1_dev"
     :param run_id_specific_config: Mapping of run_id specific config
     :param run_without_proper_corrections: Whether to run without proper
         corrections, defaults to False
@@ -174,7 +174,7 @@ def xenonnt_salted_fuse(
             "corrections for testing or just trying out fuse, "
             "set run_without_proper_corrections to True"
         )
-    if simulation_config_file is None:
+    if simu_config_version is None:
         raise ValueError("Specify a simulation configuration file")
 
     if run_without_proper_corrections:
@@ -204,6 +204,7 @@ def xenonnt_salted_fuse(
     if corrections_version is not None:
         st.apply_xedocs_configs(version=corrections_version)
 
+    simulation_config_file = "fuse_config_nt_{:s}.json".format(simu_config_version)
     fuse.context.set_simulation_config_file(st, simulation_config_file)
 
     local_versions = st.config
@@ -272,7 +273,7 @@ def xenonnt_salted_wfsim(
     corrections_version=DEFAULT_XEDOCS_VERSION,
     cut_list=cutax.BasicCuts,
     auto_register_cuts=True,
-    faxconf_version="sr0_v4",
+    simu_config_version="sr0_v4",
     cmt_version="global_v9",
     cmt_run_id="026000",
     generator_name="flat",
@@ -293,8 +294,8 @@ def xenonnt_salted_wfsim(
     :param cut_list: Cut list to use, defaults to cutax.BasicCuts
     :param auto_register_cuts: Whether to automatically register cuts,
         defaults to True
-    :param faxconf_version: (for simulation) fax configuration version
-        to use, defaults to "sr0_v4"
+    :param simu_config_version: (for simulation) fax configuration
+        version to use, defaults to "sr0_v4"
     :param cmt_version: (for simulation) CMT version to use, defaults to
         "global_v9"
     :param cmt_run_id: (for simulation) CMT run ID to use, defaults to
@@ -327,7 +328,7 @@ def xenonnt_salted_wfsim(
             print(f"Instructions saved to {instr_file_name}")
 
     # Based on cutax.xenonnt_sim_base()
-    fax_conf = "fax_config_nt_{:s}.json".format(faxconf_version)
+    fax_conf = "fax_config_nt_{:s}.json".format(simu_config_version)
 
     # Based on straxen.contexts.xenonnt_online()
     if kwargs is not None:
@@ -422,7 +423,7 @@ def fxenonnt(
     output_folder="./fuse_data",
     cut_list=cutax.BasicCuts,
     corrections_version=DEFAULT_XEDOCS_VERSION,
-    simulation_config_file="fuse_config_nt_sr1_dev.json",
+    simu_config_version="sr1_dev",
     run_id_specific_config={
         "gain_model_mc": "gain_model",
         "electron_lifetime_liquid": "elife",
@@ -448,19 +449,16 @@ def fxenonnt(
         with cutax latest
     :param cut_list: List of cuts to register, default is
         cutax.BasicCuts
-    :param auto_register_cuts: Whether to auto register cuts, default
-        True
-    :param faxconf_version: fax config version to use, default is synced
-        with cutax latest
-    :param cmt_version: cmt version to use, default is synced with cutax
-        latest
-    :param cmt_run_id: cmt run id to use, default is synced with cutax
-    :param generator_name: (for simulation) Instruction mode to use,
-        defaults to 'flat'
-    :param recoil: (for simulation) NEST recoil type, defaults to 7
-        (beta ER)
+    :param simu_config_version: fax config version to use, default is
+        synced with cutax latest
+    :param run_id_specific_config: Mapping of run_id specific config
+    :param run_without_proper_corrections: Whether to run without proper
+        corrections, defaults to False
+    :param generator_name: Instruction mode to use, defaults to 'flat'
+    :param recoil: NEST recoil type, defaults to 8 (beta ER)
     :param simu_mode: 's1', 's2', or 'all'. Defaults to 'all'
-    :param kwargs: Additional kwargs to pass
+    :param kwargs: Extra options to pass to strax.Context or generator,
+        and rate for generator
     :return: strax context
     """
     assert saltax_mode in SALTAX_MODES, "saltax_mode must be one of %s" % (SALTAX_MODES)
@@ -477,7 +475,7 @@ def fxenonnt(
         output_folder=output_folder,
         corrections_version=corrections_version,
         cut_list=cut_list,
-        simulation_config_file=simulation_config_file,
+        simu_config_version=simu_config_version,
         run_id_specific_config=run_id_specific_config,
         run_without_proper_corrections=run_without_proper_corrections,
         generator_name=generator_name,
@@ -494,7 +492,7 @@ def sxenonnt(
     corrections_version=DEFAULT_XEDOCS_VERSION,
     cut_list=cutax.BasicCuts,
     auto_register_cuts=True,
-    faxconf_version="sr0_v4",
+    simu_config_version="sr0_v4",
     cmt_version="global_v9",
     cmt_run_id="026000",
     generator_name="flat",
@@ -517,8 +515,8 @@ def sxenonnt(
         cutax.BasicCuts
     :param auto_register_cuts: Whether to auto register cuts, default
         True
-    :param faxconf_version: fax config version to use, default is synced
-        with cutax latest
+    :param simu_config_version: fax config version to use, default is
+        synced with cutax latest
     :param cmt_version: cmt version to use, default is synced with cutax
         latest
     :param cmt_run_id: cmt run id to use, default is synced with cutax
@@ -545,7 +543,7 @@ def sxenonnt(
         corrections_version=corrections_version,
         cut_list=cut_list,
         auto_register_cuts=auto_register_cuts,
-        faxconf_version=faxconf_version,
+        simu_config_version=simu_config_version,
         cmt_version=cmt_version,
         cmt_run_id=cmt_run_id,
         generator_name=generator_name,

diff --git a/saltax/instructions/README.md b/saltax/instructions/README.md
@@ -1,6 +1,6 @@
 # Instructions
 ## Scope
-`wfsim` instructions.
+Generator for simulation instructions in `wfsim` format. If you specify `fuse` as your simulation package, it will be translated when you actually compute `microphysics_summary`.
 ## Structure
 - `generator.py`: `wfsim` instruction infrastructures and instruction generators.
 ## Available Generators

diff --git a/saltax/match/README.md b/saltax/match/README.md
@@ -2,6 +2,6 @@
 ## Scope
 Functionality for matching salt and simu, with some regular follow-up analysis functions.
 ## Structure
-`match.py`: core functions for matching simulation and sprinkled dataset. The core functions to use are `match_peaks` or `match_events`.
-`utils.py`: useful functions for analysis after matching is finished. Check tutorial notebooks for usage.
-`visual.py`: a toy non-interactive event viewer. The core functinos to use are `plot_event_wf_w_data` or `plot_event_wf_wo_data`. (The only discrepancy is whether you want to show data-only reconstruction result)
+- `match.py`: core functions for matching simulation and sprinkled dataset. The core functions to use are `match_peaks` or `match_events`.
+- `utils.py`: useful functions for analysis after matching is finished. Check tutorial notebooks for usage.
+- `visual.py`: a toy non-interactive event viewer. The core functinos to use are `plot_event_wf_w_data` or `plot_event_wf_wo_data`. (The only discrepancy is whether you want to show data-only reconstruction result)