Skip to content

Latest commit

 

History

History
123 lines (73 loc) · 6.54 KB

README.md

File metadata and controls

123 lines (73 loc) · 6.54 KB
Raw

oxkat

"Ceci n'est pas une pipe[line]"
The Treachery of Images, René Magritte, 1929

What is this?

  • A set of Python scripts with the aim of (semi-)automatically processing MeerKAT data.

  • At the core is a set of functions that generate calls to various pieces of radio astronomy software, a semi-modular bunch of CASA scripts for performing reference calibration, and a fairly sizeable list of default parameters. The defaults at present cater for full-band Stokes-I continuum imaging, including the correction of direction-dependent effects.

  • Job script generation and dependency chains are automatically handled when running on the ilifu/IDIA cluster, UKZN's hippo cluster, or the CHPC's Lengau cluster.

  • Setup scripts glue the above components together into a processing recipe. The default procedure is broken down into stages, after each of which it is advisable to pause and examine the state of the process before continuing.

  • The intention is that the bar to entry is low. If you have stock Python then nothing else needs installing apart from Singularity, which is available on both the ilifu/IDIA and CHPC clusters. All the underlying radio astronomy packages are containerised. The Singularity layer can also be disabled for running installations on your own machine, either directly, or inside a Python virtual environment.

  • If you publish results that have made use of oxkat then please cite the ACSL entry, and (more importantly) the underlying packages used.

  • Please file bugs, suggestions, questions, etc. as issues.

Quick start

  1. Once you have the container(s) in place then log into your machine or cluster, e.g.:

    $ ssh [email protected]

  2. Navigate to a working area / scratch space:

    $ cd /scratch/users/ianh/XMM12

  3. Clone the root contents of this repo into it:

    $ git clone https://github.com/IanHeywood/oxkat.git .

  4. Make a symlink to your MeerKAT Measurement Set (or place it in the working folder, it will not be modified at all):

    $ ln -s /idia/projects/mightee/1538856059/1538856059_sdp_l0.full_1284.full_pol.ms .

  5. The first step is to run a script that gathers some required information about the observation:

    $ python setups/0_GET_INFO.py idia
$ ./submit_info_job.sh

  6. Once this is complete you can generate and submit the jobs required for the reference calibration (1GC):

    $ python setups/1GC.py idia
$ ./submit_1GC_jobs.sh

  7. If something goes wrong you can kill the running and queued jobs on a cluster with:

    $ source SCRIPTS/kill_1GC_jobs.sh

  8. Once all the jobs have completed then you can examine the products, and move on to the setup for the next steps in the same fashion.

Please see the setups README for more details about the general workflow. Most of the settings can be tuned via the config.py file.

Containers

There is a dedicated Singularity container (oxkat-0.41.sif) available that contains all the necessary packages and dependencies. This is available in the general container repository on the ilifu/IDIA cluster, and the default settings should pick it up automatically when that cluster is being used. For other systems the container will have to be downloaded (or copied over). The container can be downloaded here.

String patterns for package-specific containers are specified in the config.py file. The scripts will search for containers that match these patterns in the container paths, so it's simple to swap a particular package out for a different version as long as you have it containerised.

Software package roll-call

Package Stage Purpose Reference
astropy 1GC, 3GC Coordinates, time standards, FITS file manipulation Astropy Collaboration, 2013, Astropy Collaboration, 2018
CASA 1GC Averaging, splitting, cross calibration, DI self-calibration, flagging McMullin et al., 2007
CubiCal 2GC, 3GC DI / DD self-calibration Kenyon et al., 2018
DDFacet 3GC Imaging with direction-dependent corrections Tasse et al., 2018
killMS 3GC DD self-calibration Tasse, 2014; Smirnov & Tasse, 2014
owlcat 2GC, 3GC FITS file manipulation -
ragavi 1GC, 2GC Plotting gain solutions -
shadeMS 1GC Plotting visibilities Smirnov et al., 2022
Singularity 1GC, FLAG, 2GC, 3GC Containerisation Kurtzer, Sochat & Bauer, 2017
tricolour FLAG Flagging Hugo et al., 2022
wsclean FLAG, 2GC, 3GC Imaging, model prediction Offringa et al., 2014

Thank you for visiting.