Clip negative FCI radiances #3013

gerritholl · 2024-12-09T15:43:47Z

Optionally clip negative FCI radiances.

Work in progress. So far, only failing imagery tests added.

Closes artefacts in FCI RGBs using 3.8 µm #3009
Tests added
Fully documented

codecov · 2024-12-09T15:49:54Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.10%. Comparing base (5984c29) to head (4c912eb).
Report is 3 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3013   +/-   ##
=======================================
  Coverage   96.10%   96.10%           
=======================================
  Files         377      377           
  Lines       55163    55163           
=======================================
  Hits        53012    53012           
  Misses       2151     2151

Flag	Coverage Δ
behaviourtests	`3.94% <ø> (ø)`
unittests	`96.19% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

coveralls · 2024-12-09T16:03:30Z

Pull Request Test Coverage Report for Build 12318583993

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 96.206%

Totals
Change from base Build 12227072168:	0.0%
Covered Lines:	53256
Relevant Lines:	55356

💛 - Coveralls

gerritholl

start behave test

rymdulf · 2024-12-13T11:02:30Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T11:04:38Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl · 2024-12-13T11:08:13Z

rymdulf is lying, the tests were not successful. Failed with:

  Scenario Outline: Compare generated image with reference image -- @1.1         # features/image_comparison.feature:10
    Given I have a airmass reference image file from GOES17 resampled to <area>  # features/steps/image_comparison.py:54
[ WARN:[email protected]] global loadsave.cpp:241 findDecoder imread_('/app/ext_data/reference_images/reference_image_GOES17_airmass_<area>.png'): can't open/read file: check file path/integrity
    When I generate a new airmass image file from GOES17 with abi_l1b for <area> # features/steps/image_comparison.py:64
      Traceback (most recent call last):
        File "/app/venv/lib/python3.10/site-packages/behave/model.py", line 1329, in run
          match.run(runner.context)
        File "/app/venv/lib/python3.10/site-packages/behave/matchers.py", line 98, in run
          self.func(context, *args, **kwargs)
        File "features/steps/image_comparison.py", line 82, in step_when_generate_image
          ls = scn.resample(area)
        File "/app/repository/satpy/scene.py", line 980, in resample
          self._resampled_scene(new_scn, destination, resampler=resampler,
        File "/app/repository/satpy/scene.py", line 869, in _resampled_scene
          destination_area = self._get_finalized_destination_area(destination_area, new_scn)
        File "/app/repository/satpy/scene.py", line 905, in _get_finalized_destination_area
          destination_area = get_area_def(destination_area)
        File "/app/repository/satpy/resample.py", line 231, in get_area_def
          return parse_area_file(get_area_file(), area_name)[0]
        File "/app/venv/lib/python3.10/site-packages/pyresample/area_config.py", line 147, in parse_area_file
          return _parse_yaml_area_file(area_file_name, *regions)
        File "/app/venv/lib/python3.10/site-packages/pyresample/area_config.py", line 201, in _parse_yaml_area_file
          raise AreaNotFound('Area "{0}" not found in file "{1}"'.format(area_name, area_file_name))
      pyresample.area_config.AreaNotFound: 'Area "<area>" not found in file "[\'/app/repository/satpy/etc/areas.yaml\']"'

gerritholl

start behave test

rymdulf · 2024-12-13T11:56:22Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T11:59:38Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl

start behave test

rymdulf · 2024-12-13T12:11:20Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T12:15:25Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl

start behave test

rymdulf · 2024-12-13T13:27:59Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T13:32:00Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl

start behave test

rymdulf · 2024-12-13T13:43:41Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T13:47:11Z

The testing process was executed successfully. See the test results for this pull request here!

Trying John Cintineos workaround.

gerritholl

run behave test

gerritholl

start behave test

rymdulf · 2024-12-13T14:59:53Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T15:02:46Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl · 2024-12-13T15:14:21Z

The testing process keeps failing. Despite import hdf5plugin, reading FCI data fails:

    When I generate a new cloudtop image file from Meteosat-12 with fci_l1c_nc for sve # features/steps/image_comparison.py:69
      Traceback (most recent call last):
        File "/app/venv/lib/python3.10/site-packages/behave/model.py", line 1329, in run
          match.run(runner.context)
        File "/app/venv/lib/python3.10/site-packages/behave/matchers.py", line 98, in run
          self.func(context, *args, **kwargs)
        File "features/steps/image_comparison.py", line 92, in step_when_generate_image
          ls.save_datasets(writer="simple_image", filename=generated_image_path)
        File "/app/repository/satpy/scene.py", line 1296, in save_datasets
          return writer.save_datasets(dataarrays, compute=compute, **save_kwargs)
        File "/app/repository/satpy/writers/__init__.py", line 756, in save_datasets
          return compute_writer_results([results])
        File "/app/repository/satpy/writers/__init__.py", line 594, in compute_writer_results
          da.compute(delayeds)
        File "/app/venv/lib/python3.10/site-packages/dask/base.py", line 660, in compute
          results = schedule(dsk, keys, **kwargs)
        File "src/netCDF4/_netCDF4.pyx", line 5055, in netCDF4._netCDF4.Variable.__getitem__
        File "src/netCDF4/_netCDF4.pyx", line 4631, in netCDF4._netCDF4.Variable.shape.__get__
        File "src/netCDF4/_netCDF4.pyx", line 3786, in netCDF4._netCDF4.Dimension.__len__
        File "src/netCDF4/_netCDF4.pyx", line 2164, in netCDF4._netCDF4._ensure_nc_success
      RuntimeError: NetCDF: HDF error

gerritholl

start behave test

rymdulf · 2024-12-13T15:20:19Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T15:23:27Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl

start behave test

rymdulf · 2024-12-13T15:37:21Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T15:40:15Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl

start behave test

rymdulf · 2024-12-13T15:48:15Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T15:51:31Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl

start behave test

rymdulf · 2024-12-13T15:56:44Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T16:00:02Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl

start behave test

rymdulf · 2024-12-13T18:21:55Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T18:21:59Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl

start behave test

rymdulf · 2024-12-13T18:25:58Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T18:26:01Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl

start behave test

rymdulf · 2024-12-13T18:26:39Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T18:27:14Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl

start behave test

rymdulf · 2024-12-13T18:31:29Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T18:31:52Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl

start behave test

rymdulf · 2024-12-13T18:33:56Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T18:37:21Z

The testing process was executed successfully. See the test results for this pull request here!

Started behave test for FCI cloudtop

576e941

gerritholl added 3 commits December 9, 2024 18:57

Continue development of FCI imagery tests

d4a1e20

Change reference data and area for ABI test

1f562ae

Improve reference data generation script.

2297ee7

gerritholl commented Dec 13, 2024

View reviewed changes

Fix feature syntax and filename setup

02989b9

gerritholl commented Dec 13, 2024

View reviewed changes

import hdf5plugin first despite isort

8ce9a62

gerritholl commented Dec 13, 2024

View reviewed changes

Does it help if I do Meteosat-12 first?

7cd8858

gerritholl commented Dec 13, 2024

View reviewed changes

workaround for failing hdf5lpguni?

4d13d12

Trying John Cintineos workaround.

gerritholl commented Dec 13, 2024

View reviewed changes

switch on debug logging for troubleshooting purposes

4c912eb

gerritholl commented Dec 13, 2024

View reviewed changes

Clip negative FCI radiances #3013

Are you sure you want to change the base?

Clip negative FCI radiances #3013

Conversation

gerritholl commented Dec 9, 2024

codecov bot commented Dec 9, 2024 • edited Loading

Codecov Report

coveralls commented Dec 9, 2024 • edited Loading

Pull Request Test Coverage Report for Build 12318583993

Warning: This coverage report may be inaccurate.

Details

💛 - Coveralls

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl commented Dec 13, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

Choose a reason for hiding this comment

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl commented Dec 13, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

codecov bot commented Dec 9, 2024 •

edited

Loading

coveralls commented Dec 9, 2024 •

edited

Loading