code to run ngmix on data
Much of this code was written by Matt Becker based on my original gmix_meds
code base. I'm still learning it and adapting it - ESS
Please see the wiki for detailed documentation
- imageio - sub-package to read images/MEDS
- megamixer - sub-package to setup runs
- fitting.py - base class for implementing fitters
- bootfit.py - child fitter class for ngmix bootstrappers
- ngmixing.py - base class for running fitter over data (uses fitters and imageio classes)
- mofngmixing.py - MOF for running fitter over data
- nbrsfofs.py - support utils for constructing FoFs for MOF
- priors.py - code to deal with ngmix priors (doesn't really belong here, but it had to go somewhere)
- util.py - utils of various kinds
- defaults.py - default flags. etc.
- files.py - I/O utils
- render_ngmix_nbrs.py - code to render nbrs for ngmix MOF
You will need to make sure you have an appropriate MEDS I/O class.
The doc string for the base class ImageIO indicates how to write a child class that implements the
proper interface. In general, the ImageIO classes are iterators that yield a list of things to fit.
For DES MEDS files, interfaces have already been written. These are in imageio/desmedsio.py. The Y1 class
should work out-of-the box (called Y1DESMEDSImageIO).
WARNING: We have not updated the image flags for Y1. See code here https://github.com/esheldon/ngmixer/blob/master/ngmixer/imageio/desmedsio.py#L334. This needs to be done.
Whenever you implement a new interface, you have to register it in the __init__.py in the imageio sub-package.
Similar to the ImageIO classes above, you need to have an appropriate fitter.
The doc strings on all of the methods of BaseFitter in fitting.py describe what needs to be implemented.
The ngmix fitters are in bootfit.py. There are some max-like and metacal fitters there
already. Any new fitter has to be registered in the main package __init__.py.
Once you have a fitter and an imageio class, you can run the code. You will need any configs etc. setup properly.
For ngmix, see the metacal and MOF examples on the main repo wiki. Both of these examples eventally use the
ngmixit command to run the code.
For large runs, it makes sense to use the megamixer, but is not needed. The megamixer parallelizes the run by dividing
each input file to ngmixit into a bunch of smaller chunks (typically ~300 per coadd tile in DES). Job scripts and
configs are written for each of the ~300 chunks. The megamixer can then submit these jobs to the queue and recombine the
output when the jobs are done. It can also do things like tar up the log files to archive runs and collect the final
outputs into a series of flat files.
In order to parallelize the run, the megamixer uses the --fof-range input option to ngmixit to specify ranges of
objects in each file to fit. If you are not doing a MOF run, this parameter defaults to one object per FoF
(i.e., it makes dummy FoFs). Unfortunately, this whole scheme currently requires that MEDS files be input into the code.
Fixing this dependence def needs to happen, but has not yet.
To call the megamixer, use the command line util megamixit. Again see the wiki for details.
Implementing a new megamixer is not that hard at all. See the base class here
https://github.com/esheldon/ngmixer/blob/master/ngmixer/megamixer/megamixer.py
You need to supply implementations of write_job_script and run_chunk. A default
implementation is in the file above. Note that the jobs must be run in the directory
given by dr = self.get_chunk_output_dir(files,chunk,rng). The megamixer uses relative paths so that the whole set
of chunks can be relocated. Finally register the new megamixer here (https://github.com/esheldon/ngmixer/blob/master/bin/megamixit#L66).
The SLAC megamixer, which is a good example, is here
https://github.com/esheldon/ngmixer/blob/master/ngmixer/megamixer/slacmegamixer.py
Be sure to process the extra_commands option. One can use this to supply a path to the
activate function on, e.g., virtualenvs. For example, an extra_cmds.txt might look like
source /nfs/slac/des/fs1/g/sims/beckermr/DES/y1_ngmix/tb_y1a1_v01c/work011/bin/activateThe multi-object fitting (MOF) built into ngmixer is contained in the file mofngmixing.py and is iterative. For each
object on each iteration, its nbrs are subtracted and the object is refit. The ImageIO class is responsible for
yeilding all objects which need to be fit together as an entire group. The fitting class is responsible for actually
rendering the nrbs, removing them and then fitting the central.
Thus for the multi-object fitting (MOF), an extra, pre-processing step has to be taken where lists of nbrs for
each object are built. Plus other things need to be changed in the ImageIO and fitting classes to support the
propagation of the nbrs information and fit parameters.
You need to make the nbrs files for each coadd tile. This file is built off of the MEDS files made for each coadd tile and is used to decide which objects have nbrs to be fit and which ones do not.
The code which builds the nbrs files is located in nbrsfofs.py. The code works by first constructing for each a object
(i.e., the central object) a list of nbrs which 'overlap' with the central. For the MEDS files, it is assumed that the
stamp sizes themsselves indicate the extent of each object and so the overlaps are built by intersecting the stamps with
some buffer. Once each object has a list of nbr objects, a FoF group of objects is built by linking objects if either is
in the nbrs list of the other. Finally, the fits of the objects are done by fitting all things in a FoF together and
parallelizing the entire run over the set of FoFs. (For the DES, including isolated objects as FoFs of size one,
typically only 40% of the FoFs have two or more objects.)
The command line utility for doing these steps is ngmixer-meds-make-nbrs-data. It needs a config file similar to
the one here https://github.com/esheldon/ngmix-y1-config/blob/master/nbrs_config/nbrs001.yaml. See the help menu of
ngmixer-meds-make-nbrs-data for how to run it. Two nbrs files get written for each coadd tile in the $DESDATA
area and are for example
${DESDATA}/EXTRA/meds/${meds_version}/nbrs-data/${nbrs_version}/${coadd_runn}/\
${tile}-meds-${meds_version}-nbrslist-${nbrs_version}.fits
${DESDATA}/EXTRA/meds/${meds_version}/nbrs-data/${nbrs_version}/${coadd_runn}/\
${tile}-meds-${meds_version}-nbrsfofs-${nbrs_version}.fits
These are for the Y1 pipeline. The Y2+ pipeline may be different. The first file above is the list of nbrs and the second file above is the list of FoF membership constructed from the nbrs.
The ImageIO interface used for reading the image data has to be aware of the nbrs files made above and process them so
that they include the needed information in the meta_data attached to the images in order to run the MOF fits. All of
this is already done for MEDS files. See the doc string on the base ImageIO class (mentioned above) in order to see
what needs to be added if you implement a new interface.
Any fitting class with the MOF is responsible for rendering the nbrs for each central and subtracting them out of the centrals stamps. For ngmix-based MOF, this has already been done. If you are using another fitter, you will have to code this up yourself. Again see the base fitter docs for how to do this.
Once the two files above have been built, the code can be run. For the MOF the two files above have to be specified to
ngmixit with the --nbrs-file (for the nbrslist) and the --fof-file (for the nbrsfofs). If you are using the
megamixer, then this all gets done for you. You will also need to set the following in the fitting config fed to
ngmixit (assuming you are using ngmix)
read_wcs: True
read_me_wcs: False
model_nbrs: True
model_nbrs_method: 'subtract'
mof:
maxabs_conv: [1.0e-3,1.0e-3,1.0e-6,1.0e-6,1.0e-6,1.0e-6,1.0e-6,1.0e-6,1.0e-6]
maxfrac_conv: [1.0e-6,1.0e-6,1.0e-6,1.0e-6,1.0e-6,1.0e-3,1.0e-3,1.0e-3,1.0e-3]
maxerr_conv: 0.5
max_itr: 15
min_itr: 10
min_useg_itr: 1
write_convergence_data: False
convergence_model: 'cm'
prop_sat_starpix: True
flag_y1_stellarhalo_masked: True
intr_prof_var_fac: 0.0
region: "mof"This config setting is for Y1. The prop_sat_starpix and flag_y1_stellarhalo_masked options may
not apply to future data releases.
See the repo wiki.