Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tutorial for mono dl1b to dl2 and update docs #2691

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Add tutorial for mono dl1b to dl2 and update docs #2691

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
059d19a
Add tutorial for mono dl1b to dl2; update tools api docs and user-guide
LukasBeiske Jan 24, 2025
cd83c31
Address comments; add changelog
LukasBeiske Jan 27, 2025
4a1e033
Remove trailing whitespace
LukasBeiske Feb 5, 2025
5370255
Shorten reconstructor names
LukasBeiske Feb 6, 2025
5c742e8
Fix note about stereo reco
LukasBeiske Feb 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 16 additions & 4 deletions docs/api-reference/tools/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,16 +82,22 @@ Reference/API
=============

.. automodapi:: ctapipe.tools
:no-inheritance-diagram:
:no-inheritance-diagram:

.. automodapi:: ctapipe.tools.quickstart
:no-inheritance-diagram:

.. automodapi:: ctapipe.tools.process
:no-inheritance-diagram:
:no-inheritance-diagram:

.. automodapi:: ctapipe.tools.merge
:no-inheritance-diagram:

.. automodapi:: ctapipe.tools.train_energy_regressor
:no-inheritance-diagram:
:no-inheritance-diagram:

.. automodapi:: ctapipe.tools.train_particle_classifier
:no-inheritance-diagram:
:no-inheritance-diagram:

.. automodapi:: ctapipe.tools.train_disp_reconstructor
:no-inheritance-diagram:
Expand All @@ -104,3 +110,9 @@ Reference/API

.. automodapi:: ctapipe.tools.compute_irf
:no-inheritance-diagram:

.. automodapi:: ctapipe.tools.dump_instrument
:no-inheritance-diagram:

.. automodapi:: ctapipe.tools.calculate_pixel_stats
:no-inheritance-diagram:
4 changes: 4 additions & 0 deletions docs/changes/2691.maintenance.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Add a tutorial to the docs on how to use the machine learning tools
to process dl1b (image parameters) files to dl2 for a mono analysis.

Perform minor updates to other parts of the api docs and the user-guide.
134 changes: 0 additions & 134 deletions docs/user-guide/data_models/dl1.rst
View file
Open in desktop

This file was deleted.

125 changes: 123 additions & 2 deletions docs/user-guide/data_models/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,127 @@
Data Models
***********

.. toctree::
The ``ctapipe`` output files are HDF5 format files, with the following data set hierarchy.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better if you say the data hierarchy is described "below" as that implies some distance, while "following" means something more like "directly after here".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All of this is the old content of user-guide/data_models/dl1.rst. I did not write any of this, I just fixed the file structure.
I'll have a look again, if I have time.

But this would probably make more sense in a separate PR with a general overhaul of the docs.

The tables are written with `pytables <https://www.pytables.org>`_ (not ``h5py``).
Containers marked with a ``+`` should be written without their prefix (all others should use column prefixes).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not clear to me what the purpose of these lines is in the user guide? Could you clarify what you want the user to understand?


dl1
The following describes the contents of data level 1 (DL1) output files
generated by ctapipe (e.g. the ``ctapipe-process`` tool which uses the
:py:class:`~ctapipe.io.DataWriter` component to generate output data).


DL1 Data Model
==============

This describes data that change per-event.
The following datasets will be written to the group ``/dl1/event/`` in the output file:

.. list-table::
:header-rows: 1

* - Group/Dataset
- Description
- Contents
* - ``subarray``
- event-wise data pertaining to a subarray
- (group)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should probably explain what "(group)" is supposed to mean, you use with HDF5 terminology without explaining it as far as I can see.

* - ``subarray/trigger``
- subarray trigger information
- :py:class:`~ctapipe.containers.EventIndexContainer` +, :py:class:`~ctapipe.containers.TriggerContainer`
* - ``telescope/``
- Per-telescope Per-event information
- (group)
* - ``telescope/parameters/tel_{TEL_ID:03d}``
- tables of image parameters (one per telescope)
- :py:class:`~ctapipe.containers.TelEventIndexContainer` +, :py:class:`~ctapipe.containers.ImageParametersContainer`
* - ``telescope/images/tel_{TEL_ID:03d}``
- tables of telescope images (one per telescope)
- :py:class:`~ctapipe.containers.TelEventIndexContainer` +, :py:class:`~ctapipe.containers.DL1CameraContainer`


DL2 Data Model
==============

This describes data that change per-event. The following datasets will be
written to the group ``/dl2/event/stereo/<algorithm>/`` and or
``/dl2/event/mono/<algorithm>/``, one for each reconstruction algorithm in the
output file, where ``<algorithm>`` is the identifier of the algorithm (e.g.
"Hillas"):

.. list-table::
:widths: 25 50 25
:header-rows: 1

* - Group/Dataset
- Description
- Contents
* - /geometry
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here you give the group with a initial "/" but otherwise not typeset in any special way, where before there was no initial slash but the key was set as code. Later you set the groups as code and include an initial "/".

Decide on how you want to present the subgroups and implement it uniformly. I suggest the first style, set as code and without the slash.

- shower geometry reconstruction
- :py:class:`~ctapipe.containers.EventIndexContainer`, :py:class:`~ctapipe.containers.ReconstructedGeometryContainer`
* - /energy
- shower energy reconstruction
- :py:class:`~ctapipe.containers.EventIndexContainer`, :py:class:`~ctapipe.containers.ReconstructedEnergyContainer`
* - /classification
- shower classification parameters
- :py:class:`~ctapipe.containers.EventIndexContainer`, :py:class:`~ctapipe.containers.ParticleClassificationContainer`


Simulation Data Model
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this not presented as a table when all the other data models are?

=====================

* - ``/simulation/event/subarray/shower``
- true shower parameters from Monte-Carlo simulation
- :py:class:`~ctapipe.containers.EventIndexContainer` +, :py:class:`~ctapipe.containers.SimulatedShowerContainer`
* - ``/simulation/event/telescope/images/tel_{TEL_ID:03d}``
- simulated camera images
- :py:class:`~ctapipe.containers.EventIndexContainer` +, :py:class:`~ctapipe.containers.SimulatedCameraContainer`
* - ``/simulation/event/telescope/parameters/tel_{TEL_ID:03d}``
- Parameters derived form the simulated camera images
- :py:class:`~ctapipe.containers.EventIndexContainer` +, :py:class:`~ctapipe.containers.ImageParametersContainer`
* - ``/simulation/service/shower_distribution``
- simulated shower distribution histograms
- :py:class:`~ctapipe.containers.SimulatedShowerDistribution`


Configuration Data Model
========================

The output file should also contain serializations of the instrument, observation (if
applicable), simulation (if applicable) configuration information, written to the
``/configuration`` group:

.. list-table::
:widths: 25 50 25
:header-rows: 1

* - Group/Dataset
- Description
- Contents
* - ``/instrument``
- Serialized :py:class:`ctapipe.instrument.SubarrayDescription`, using :py:meth:`ctapipe.instrument.SubarrayDescription.to_hdf`
- (group)
* - ``/instrument/subarray/layout``
- Subarray layout info
- result of ``SubarrayDescription.to_table()`` output as HDF5 using ``astropy.table`` functionality
* - ``/instrument/telescope/optics``
- telescope optics information
- result of ``SubarrayDescription.to_table(kind='optics')`` output as HDF5 using
``astropy.table`` functionality
* - ``/instrument/telescope/camera/{CAMERA_ID}``
- camera geometry information
- result of ``CameraGeometry.to_table()`` output as HDF5 using ``astropy.table``
functionality
* - ``/simulation``
- Monte-Carlo simulation configuration information
- (group)
* - ``/simulation/run``
- Monte-Carlo simulation run information
- :py:class:`~ctapipe.containers.SimulationConfigContainer`


Core Provenance
===============

The root group of the file shall contain all of the "CTA Core Provenance Metadata"
headers as user attributes, with the hierarchy flattened and separated by spaces
(e.g. ``"CTA ACTIVITY NAME" = "ctapipe-process"``).
2 changes: 1 addition & 1 deletion docs/user-guide/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ User Guide
:maxdepth: 2

getting-started
tools
tools/index
data_models/index
examples_tutorials
FAQ
36 changes: 0 additions & 36 deletions docs/user-guide/tools.rst
View file
Open in desktop

This file was deleted.

Loading