Skip to content
This repository has been archived by the owner on Feb 26, 2020. It is now read-only.

Fix writer holding to memory after processing #43

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

Fix writer holding to memory after processing #43

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
c4f44fb
Delete processed data when finished writing
mraspaud Feb 8, 2019
f0592ae
Close dask's pool to free up memory after computation
mraspaud Feb 11, 2019
6bf7e72
Fix dask pool closing and add garbage collection
mraspaud Feb 11, 2019
fe7c36e
Quit writer when all files have been processed
mraspaud Feb 13, 2019
314eef6
Fix exiting satpy writer
mraspaud Feb 13, 2019
3a08f9d
Add a sleep to make sure the writer is ready before quitting
mraspaud Feb 14, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions trollflow_sat/satpy_writer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,8 @@ def run(self):
self._process(data, **kwargs)
except Exception:
self.logger.exception("Something went wrong when writing.")

finally:
del data
# After all the items have been processed, release the
# lock for the previous worker

Expand Down Expand Up @@ -222,7 +223,7 @@ def _process(self, data, **kwargs):
fmts = utils.get_format_settings(product_config, prod,
scn_metadata["area_id"])

# Read writer specific kwargs
# Read writer specific kwargs
writer_kwargs = utils.read_writer_config(product_config, products[prod], prod, scn_metadata)
kwargs.update(writer_kwargs)

Expand Down