Fix docs building and deployment

[juanep97]

It was not deploying only the latest tagged release which was the intended behavior in #91.

Notebooks were also not properly run because the data was not available in the DB. Added a step of adding the test data and reducing it before building the docs.

This step can be removed (making doc building faster) if the notebooks are modified to first make sure that local data is added to the DB (e.g. in another notebook); but this works for now.

Also, omit the docs step in ci.yml for tag pushes, in that case it is done already in docs.yml.

[morcuended]

Ups!, It was indeed helpful to check the docs: https://juanep97.github.io/iop4/recipes/05_quad_matching_dipol_polarimetry_images.html

There are some steps of the examples that failed (see page above):

DoesNotExist: RawFit matching query does not exist.

[juanep97]

Yep, I just noticed. It is missing a iop4 --list-local to reduce the test dataset and add it to the DB, before it is used in the notebooks. Fixing it.

[morcuended]

might need to create the directory first /home/runner/.iop4data/logs?

FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/.iop4data/logs/iop4.log'

[juanep97]

might need to create the directory first /home/runner/.iop4data/logs?

FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/.iop4data/logs/iop4.log'

I changed the default location to .iop4data/iop4.log, no need to have a separate folder for only one file, I think.

[juanep97]

might need to create the directory first /home/runner/.iop4data/logs?

FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/.iop4data/logs/iop4.log'

I changed the default location to .iop4data/iop4.log, no need to have a separate folder for only one file, I think.

I changed my mind... The online log viewer assumes that log files are at {{datadir}}/logs/, so better to create the dir anyway if needed during configuration, otherwise the user might encounter an error when using the web interface for the first time.

[juanep97]

iop4/iop4lib/db/masterdark.py

Lines 189 to 201 in a501c3b

	def build_file(self):
	logger.debug(f"Getting data from files")
	if self.rawfits.count() == 0:
	raise Exception(f"No rawfits for {self}")
	data_L = []
	for rf in self.rawfits.all():
	with fits.open(rf.filepath) as hdul:
	data = (hdul[0].data - self.masterbias.data) / self.exptime
	data_L.append(data)
	data = np.nanmedian(data_L, axis=0)

Looks like that part uses a lot of memory (over 4GB) when it creates the master dark for those 14 files... it should not use that much. It was making docs fail. After increasing the runner limits, docs passes but it still slows down quite a bit.

Probably that np.nanmedian call.

[morcuended]

Maybe using overwrite_input=True in that nanmedian call would reduce memory usage (since data_L is no longer used)?

[juanep97]

Maybe using overwrite_input=True in that nanmedian call would reduce memory usage (since data_L is no longer used)?

I will reduce the limits to the original and re-run it with that change to test it. If it does not work, we can open a separate issue / PR to study it.

[juanep97]

Nope, it gets stuck and files there:

2024-04-11 11:03:23 - iop4lib.db.epoch [epoch.py:396] - INFO - Building Master Dark for OSN-T090/2023-10-09 | DIPOL | 4144x2822 | 30.0s.
2024-04-11 11:03:23 - iop4lib.db.masterdark [masterdark.py:144] - INFO - A MasterDark entry for OSN-T090/2023-10-09 | DIPOL | 4144x2822 | 30.0s will be created.
2024-04-11 11:03:23 - iop4lib.db.masterdark [masterdark.py:146] - DEBUG - Saving MasterDark to DB so that it has an id.
2024-04-11 11:03:23 - iop4lib.db.masterdark [masterdark.py:151] - DEBUG - Found 14 dark raw files for <MasterDark 3 | OSN-T090/2023-10-09 | DIPOL | 4144x2822 | 30.0s>.
2024-04-11 11:03:23 - iop4lib.db.masterdark [masterdark.py:161] - DEBUG - Using <MasterBias 5 | OSN-T090/2023-10-09 | DIPOL | 4144x2822> as MasterBias for <MasterDark 3 | OSN-T090/2023-10-09 | DIPOL | 4144x2822 | 30.0s>.
2024-04-11 11:03:23 - iop4lib.db.masterdark [masterdark.py:166] - INFO - Building file
2024-04-11 11:03:23 - iop4lib.db.masterdark [masterdark.py:190] - DEBUG - Getting data from files

Full frame darks for DIPOL are 23M, 14 of them should be 322M. Since they get converted to from int16 to float64 for the operations, that is 1.2GB. I assume that the median algorithm uses some factor of this, hence the memory usage. Possibly could be alleviated with something similar to #30. Also, in the close future we will increase the default binning of DIPOL to reduce file size.

[morcuended]

OK, thanks for the checks. Then, increase the runner's memory and we will deal with that problem in another PR as you suggested. This one is good to go