changelog.txt

Changelog - to keep track of all relevant changes

please update the categories "new features" and "bugfixes" before a pull request merge!

version 3.0.0-dev
new features:
- created module and files to retrieve cable and RCU responses for LOFAR
- added module to incorporate the LOFAR hardware response into the signals
- added files containing the coefficients for the LOFAR Galactic calibration module in utilities/data/galactic_calibration
- added LOFAR modules to fit the arrival direction using a planewave fit and to visualize the results of the pipeline
- added support for LBA INNER in the LOFAR reader
- added LOFAR detector description in JSON format
- cleaned up the LOFAR beamforming utilities module
- updated logging in all LOFAR modules
- added support for more Galactic noise models in channelGalacticNoiseAdder
- added support for "ant_mode" keyword in detector description
- added option to choose dipole orientation when creating the LOFAR LBA antenna pattern
- implementation of additional trigger channels. The signals on which the trigger is determined might be passed
- added 2024 detector decription file in NuRadioReco/detector/RNO_G/RNO_season_2024.json
through a different response chain as the signals which are readout. To allow simulating this different response
we added an optional trigger_channel object which lives within the corresponding channel obejct.
- Updated IC limit

bugfixes:
-

version 3.0
NuRadioMC 3.0 is a complete refactor of the NuRadioMC core (simulation.py), enabling long-awaited features such as:
- exposing functions to calculate voltage traces from showers for stand-alone use (needed e.g. for Likelihood forward folding reconstruction)
- separate simulation of trigger channels (-> significant speed up of simulations)
- allow simulation of non-consecutive channel ids
- enable resimulation from nur files (not yet implemented but prepared)
- adding additional antennas on-the-fly to existing simulations
To enable resimulation from nur files, the nur files are the master file format and contain all information of the simulation. The hdf5 files
are still generated from the nur files to have the main simulation results available in a simple table, e.g., for quick effective volume calculations.

The changes are transparent to the user. All usecode still works and produces identical results. (Only the triggerTimeAdjuster module
was integrated into the main simulation routine and is not needed anymore in the user code.)
See https://github.com/nu-radio/NuRadioMC/pull/680 for details.
To simulate trigger channels separately, a list of trigger channels ids need to be passed to the __init__ function of the simulation class.

new feature
- pre trigger times, i.e., how much trace should be saved before the trigger occured, is defined in the
trigger modules (and not implicitely through the triggerTimeAdjuster module). The trigger objects have a new flag
`primary` which defines that this trigger was used to set the readout windows (in the case of multiple triggers.)
Only one trigger can be the primary.
- BaseTrace has a new utility function `add_to_trace`, which adds a trace into an existing one.

version 2.3.0-dev
new features:
- added module to create CR templates for RNO-G channels
- added function to calculate correlation between two traces using a sliding window (with variable window size)
- Enhanced detector_browser to allow display RNO-G database detector
- Added download utility unifying download of antenna models/proposal tables/shower
library with possibility to add/change fallback-server(s). Added Chicago
server as fallback.
- Added new detector class for RNO-G (+db interface) which uses a mongo-db as source.
The new class allows import/export via compressed json files and has a buffer machinary.
It comes with a class to handle responses of the different signal chain components easily.
- Improved logging module: streamlined logging for NuRadioReco and NuRadioMC modules,
added new STATUS logging level by default, added logging documentation and moved logging
module to NuRadioReco/utilities folder. Logging functionality from NuRadioReco/modules/base
is now deprecated.
- Added StationParameters to store channel IDs of flagged (i.e.) bad channels, the polarisation
which contains the strongest CR pulse and the list of frequency bands which are flagged for RFI.
- Added interface to give (Sim)Channel a group ID . This is implemented as a keyword in the
Detector description, such that it can be added to the Detector description JSON file. The
(Sim)Channel objects can now optionally take a channel_group_id parameter. In a Station, one
can then iterate over a channel group, which yields all channels for a given group ID.
- Added Gaussian tapered and Hann tapered bandpass filters
- Added LOFAR modules for reading data, removing RFI, beamforming and fitting the arrival direction
- Added functionality to remove Channel from Station
- Added LOFAR antenna pattern and function to parse TXT simulation file
- The assume_inf and antenna_by_depth Detector keywords are now saved in the nur file, and will be
loaded again when reading in the Detector from a nur file
- Added LOFAR coordinates to Detector site coordinates
- Added up to date in-ice VPol response (RNOG_vpol_v2_5inch_center_n1.75.pkl) sha-1 hash
- Implemented mattak dataset iterator in readRNOGDataMattak.run()
- Fixed bug in phased-array noise generation script where the sign of the time delays was being ignored
for the phasing_mode == "slice"
- Improved logging: created NuRadioLogger class which is now used by default, the NRR and NRMC loggers
are created automatically when importing the packages, and a new STATUS logging level was added.
- Updating the ray_tracing_2D class in order to be able to use Numba optimization
- Add new cw filter module to NuRadioReco using notch filters on peaks in the frequency spectrum

bugfixes:
- Fixed bug in get_travel_time in directRayTracing propagation module
- Fixed minor bugs in analyticraytracing module. Path length and travel times are affected by O(0.001) ns,
  and the focusing factor is reduced by <1% for most geometries. See #649 for details.

version 2.2.1
bugfixes:
- readRNOGDataMattak: fix bug where .root files would not be found if they are passed as relative paths (instead of absolute paths or folder)

version 2.2.0
new features:
- expand values stored in SimplePhasedTrigger
- added getting to query ARZ charge-excess profiles
- upgrade to proposal v 7.5.0 with new NuRadioProposal interface and improved LPM treatment
- add script to generate / download pre-calculated proposal tables for known detector configurations
- adding default RadioPropa ice model object to medium with the feature to set a personlised object as alternative
- add positions array functionality to simple ice model in average and gradient functions
- analytic ray tracing solutions are now sorted consistently from lowest to highest ray
- added ability to generate high-low-triggered noise on a narrow band but return full-band waveforms
- phased array noise generation utility class: calculate trigger time and cut trace accordingly
- use Philox noise generator in noise adder module (this changes the default random sequence)
- allow raytracing from air to ice and vice versa. Only supported by the Python implementation. (Note that the calculation of the focussing factor was not thoroughly tested.)
- added 'block offset' removal/simulation module for RNO-G
- triggerTimeAdjuster now works more reliably and supports different readout windows per channel.
  The readout time adjustment now is stored in the trigger object and needs to be accounted for in analysis
  by running the triggerTimeAdjuster, analogously to the channelAddCableDelay module.



bugfixes:
- fixed/improved C++ raytracer not finding solutions for some near-horizontal or near-shadowzone vertices
- fixed wrong number in Feldman-Cousins upper limit
- updated reference channel check in GenericDetector to look into the reference station when the channel
is not found in the current station
- in antennapattern.py, fixed line 1398; was masking frequencies >= 0 instead of frequencies > 0, causing NaN errors
- Fixed issue where saving empty traces (channels or Efields) created unreadable .nur files

version 2.1.8
bugfixes:
- replace deprecated np.float with float

version 2.1.7
new features:
- add attenuation model from the 2021 measurements taken at Summit Station
- print warning for positive z in attenuation, and set attenuation length to +inf for positive z
- adding minimize mode to the radiopropa propagation module & more configurable settings in config file
- add uniform ice model with 1.78 refractive index
- update ARA detector description and add a file to Print ARA detector json file
- pass evt_time to sim station
bugfixes:
- fix bug in NuRadioProposal which pervented muons from tau decays to be propagated
- update proposal version to 6.1.8 to avoid problems with pypi
- updated reference channel check in GenericDetector to look into the reference station when the channel
is not found in the current station

version 2.1.6
bugfixes:
- the n_interaction parameter was accidentally overridden. n_interaction counts the number of interactions of taus and
muons generated in the corresponding CC interactions. The bug resulted in the following behaviour: The n_interaction value of the primary
neutrino interaction was overridden with the n_interaction value of the last shower of the previous event group. If the previous event group
was a CC interaction, the n_interaction value of the primary interaction was set to a value larger than 1. If that happened, and if the primary
shower did not trigger the detector, this shower is not added to the output hdf5 file. This has consquences for analyzing hdf5 files for
secondary interactions. With this bug, the initial interaction needs to be idenfified with np.abs(flavor) == 14 or 16.
This bug does not effect any effective volume calculation.

new features:
- add hvsp1, update idl1 model for emitter simulation (from KU lab measurements) and remove hvsp2(no longer in use for experiment)
- add CalPulser data for emitter simulation

version 2.1.5

bugfixes:
- the phased array trigger module did not calculate the trigger time.
  The trigger time was always set to the trace start time. This got fixed in this version.

version 2.1.4

bugfixes:
- fix passing hdf5 files in memory (generation attributes were not set which resulted in a crash)

version 2.1.3
bugfixes:
- PROPOSAL energy cut (min_energy_cut_nu) assumed wrong units (before: 1e-6*units.eV)

version 2.1.2
bugfixes:
- Fixes that the generic detector crashes for certain detector files.

version 2.1.1
new features:
- add a numerical raytracer depending on the radiopropa code
- major change in the declaration of mediums at the back end, at the front end nothing changed.
- Added neutrino direction reconstruction module and example on how to use it.
- inheritance of ray tracers form a base class
- changes to the eventbrowser in order to read RNO-G data
- add sphericalWaveFitter module
- add directory RNO_data with an example reconstruction for RNO data and the calculation of the cable delays
- add Genie HEDIS (BGR18) cross section as an optional cross sec model
- add handling of multiple reference channels and stations to generic detector
- add thermal noise generator that fulfills phased array trigger
- add a basic uproot data reader for RNO-G data
- add option to simulate emitters
- added helper function for cosmic ray flux models
- speed improvements of ARZ model through use of numba

bugfixes:
- correct volume extension depending on zenith angle range when running with
secondaries
- corrected get_hilbert_envelope(_mag) in BaseTrace to actually return the Hilbert envelope and simplified resamle function

version 2.0.1
new features:
- Veff utility can now calculate effective volumes for fine zenith angle binning
- Monte Carlo info introduced in .nur files via Particle and generator_info

bugfixes:
- fixed rare cases where the Cpp implementation of the analytic ray tracer did not find a solution



version 2.0.0
- NuRadioReco is merged into the NuRadioMC repository. No new features were added and everything works (e.g. the imports)
as before but the NuRadioReco code is now part of the NuRadioMC github repository to simplify code development where NuRadioMC
changes depend on changes in NuRadioReco.


version 1.2.0
new features:
- major change in internal looping. Now, the radio signal from each shower is calculated and signal arrival times
  (from different shower, ray racing solutions and receiver positions) that cluster together are simualted as one event
- merge hdf5 utility has multithreading option (and was refactored -> function names changed)
- distance cut can sum up all sourounding shower energies to properly account for possible interference
- noise temperature can be specified per station and channel
- trigger thresholds can be specified per channel
- bandwidth can be specified per station and channel
- specifying the detector simulation became easier (check out the updated examples)
- memory consumption was optimized to stay <4GB per core
- random realization of showers are saved so that triggered events can be resimulated using the same random realization
- added option for noiseless channels in a "with noise" simulation
- add option to generate events on the fly and pass them directly to the simulation part (no need to save input hdf5 files anymore)
- added uncertainties to CTW cross sections
-

bugfixes:
- Fixed issue with merge hdf5 utility so that "event_group_ids" are properly unique




version 1.1.2 -
new features:
- Veff utility can now handle extended bins
- New tutorial and example for the webinar
- trace start time for the electric field models adjusted such that global time of pulse position corresponds to propagation time
- Simplified treatment of reference angles and polarization for the ARZ module
- Proposal 6.1.1 supported
- Safeguard for events at more than 20 degrees from the Cherenkov angle when using the ARZ models
- Antenna model now needs to be fully specified in the detector description (previously `_InfFirn` was automatically appended to the antenna name for antennas below the surface)

bugfixes:
- Fixed primary particle code bug when using Proposal
- Fixed geometry cut when generating secondary particles
- Incorrect Veff of secondary interactions fixed

version 1.1.1 - 2020/03/23
new features
- New version for the ARZ model available (ARZ2020)
- a list with event ids can be passed to the main simulation class. All events not in this list will not be simulated.
  This is useful for a quick resimulation of certain events.
- Alvarez???? Askaryan models now place the trace into the center of the trace (instead of 50ns into the trace)
- New data set array 'vertex_times' contains the time elapsed from the first interaction to the current interaction
- new utility to split output hdf5 files into smaller chucks (to be able to resimulate events on a cluster)
- Greenland added to proposal config
- new unforced event generator module
- improved Earth model (PREM), path from interaction vertex through Earth is calculated (before interaction was assumed
  to happen at the surface)
- detector description is saved to nur output files
- new handling of random numbers. Each module has its own random generator instance. Global seed can be controlled via
  config file setting.

bugfixes:
- ARZxxxx and Alvarez2009 Askaryan modules now use the same (random) shower per event.
- fixes zenith distribution of event generator back to cos(zenith)
- ray tracing precision was increased to 1e-9
- saveguard against too many ray tracing solutions added

version 1.1.0 - 2020/01/15
new features:
- specifying the bandwidth of the detector is no longer required. Now, the bandwidth is calculated automatically from the
  detector_simulation that the user defines by examining the NuRadioReco modules that alter the bandwidth (i.e. bandpass filters
  and hardware respone incoroprator) using the new module registry feature of NuRadioReco. The noise RMS is then calculated
  from the noise temperature and the bandwidth of the detector.
  Alternatively, the user can specify the RMS noise value manually which is useful to simulate an existing detector where
  the noise RMS is known.
- production includes testing of NuRadioReco files
- NuRadioMC simulations with noise and at Moore's Bay site are tested in our CI test suite
- Event generation using PROPOSAL lepton propagation available
- fft and units utilities are removed. These should be taken from NuRadioReco from now on.
  IMPORTANT: Note that in the fft the frequency spectrum is now given per frequency instead of per bin


bugfixes:
	- fixed that Veff utility script didn't calculate effective volumes for individual triggers
	- fixed that the multiple_triggers key was not created for multiple stations
	- Angular distribution now accounts for projected area
	-


version 1.0.0 - 2019/08/30 - first python 3 release



=======
NuRadioReco changlog from previous setup where NuRadioReco was in its own repository


version 1.2
new features:
- users can specify a different bandwidth for each channel in the channelBandPassFilter and channelGenericNoiseAdder module
- new coreas read in function to generate signals for a grid of stations
- added simulation of galactic noise
- new phased array trigger module
- eventWriter: Add option to specify number of events per file

bug fixes:
- Added if check in voltageToEfieldConverter.py under method get_array_of_channels() to see if sim station is initialized
- trigger modules set the trigger time to 0 in case not trigger was recorded. This lead to problems, and wrong total trigger times, if multiple triggers were recorded. Now, no trigger time is set if not trigger was found.
- made detector_sys_uncertainties compatible with python3 and detector updates
- Typo in antennapattern.py; processing antenna responses fails. replaces s_parameteres with s_parameters


version 1.1.2 -

new features:
- Added the ability to force polarization to be only eTheta or ePhi in the reconstruction of the electric field using voltageToEfieldConverter.py
- new detector which allows to change certain parameters of the detector description to study systematic uncertainties.
- Simulations with alias (higher Nyquist zones) for the phased array now supported
- NuRadioRecoIO has a 100MB read buffer by default
- Changed detector description to optionally take antenna model name directly from detector description (without automatically selecting antenna model dependent on depth of the antenna)
- NuRadioReco now fails if no antenna pattern was found

bug fixes:
-Removed function "get_relative_positions(self, station_id)" from detector.py because the ordering of channels is different from a sql initiated detector compared to a json initiated detector. Must use "get_relative_position(self, station_id, channel_id)", which specifies which channel to get the position for.


version 1.1.1 - 2020/03/23

new features:
- changed the way logging is handled.
- RNO-G hardware response added


version 1.1.0 - 2020/01/15

new features:
- added module registry. By adding a decorator to the run methods of every module, we keep track of the modules applied to an event
-Added new module called "beamFormingDirectionFitter" that fits direction through an interferometry method (dubbed beamForming)
-Added the internal clock parameter (time since last trigger with ms precision) to ARIANNA paremeters (key number 8)
-Changed import "detector_sql" to "from NuRadioReco.detector import detector_sql" and .GetSec() to datetime.datetime.fromtimestamp(self.config_tree.TrigStartClock.GetCurrTime()) in the detector.py and readARIANNAdataCalin.py files respectively in order to make it python3 compatible.
-Updated documentation and made it deployable on github
-Changed normalization of the frequency spectrum. The amplitude is now given as V/GHz (or V/m/GHz for E-fields) instead of
   V/GHz/bin. This way, the values of trace.get_frequency_spectrum() are independent of the sampling rate. The fft utilities
   now also need the sampling rate as an argument.
-Envelope phased array available
-Trigger times now include the time with respect to the first interaction (vertex times)
-Analog to digital converter module added
-Improved calculation of the diode noise parameters
-channelSignalReconstructor no longer requires static signal window but can set it automatically
-Save the git hashes of the NuRadioReco, NuRadioMC and radiotools version a .nur file was created with

Detector description can be stored in .nur files
Large overhaul of the event structure. Adds shower classes and hybrid detector information.

bugfixes:
-Fixes increase in filesize caused by switch to python3
-Fixed bug when using no secondary channels on the phased array
-Fixed bug when using ARA trigger


version 1.0.0 - 2019/08/30 - first python 3 release