Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Correct Unexpected floats when reading LI L2 LFL #2998

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

ClementLaplace
Copy link
Contributor

Makes sure that the method apply_fill_value is not applied when _FillaValue is not into the attributes. It is fixing the bug mentioned there #2854 .

Copy link

codecov bot commented Nov 22, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.09%. Comparing base (da84875) to head (0821afb).
Report is 44 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2998      +/-   ##
==========================================
+ Coverage   96.08%   96.09%   +0.01%     
==========================================
  Files         377      377              
  Lines       55150    55167      +17     
==========================================
+ Hits        52992    53014      +22     
+ Misses       2158     2153       -5     
Flag Coverage Δ
behaviourtests 3.94% <0.00%> (-0.01%) ⬇️
unittests 96.19% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@coveralls
Copy link

coveralls commented Nov 22, 2024

Pull Request Test Coverage Report for Build 12184523570

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 21 of 21 (100.0%) changed or added relevant lines in 3 files are covered.
  • 3 unchanged lines in 3 files lost coverage.
  • Overall coverage increased (+0.01%) to 96.203%

Files with Coverage Reduction New Missed Lines %
satpy/tests/reader_tests/_li_test_utils.py 1 98.55%
satpy/readers/li_base_nc.py 1 99.62%
satpy/tests/reader_tests/test_li_l2_nc.py 1 99.53%
Totals Coverage Status
Change from base Build 11936881763: 0.01%
Covered Lines: 53258
Relevant Lines: 55360

💛 - Coveralls

Copy link
Member

@ameraner ameraner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the investigations! I left some comments in the code.

I am also wondering if we still also need the fix for the float64 upcasting when multiplying with the scale_factor, as discussed in the issue...

satpy/readers/li_base_nc.py Outdated Show resolved Hide resolved
satpy/readers/li_base_nc.py Outdated Show resolved Hide resolved
satpy/tests/reader_tests/test_li_l2_nc.py Show resolved Hide resolved
…tests to check the dtype improve the mock datas
Copy link
Member

@ameraner ameraner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix and the extra more comprehensive tests! Just two things inline.

@@ -306,7 +306,7 @@ def get_projection_config(self):
"""Retrieve the projection configuration details."""
# We retrieve the projection variable name directly from our swath settings:
proj_var = self.swath_coordinates["projection"]

logger.error(proj_var)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this one should be removed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I have removed this

satpy/readers/li_l2_nc.py Outdated Show resolved Hide resolved
satpy/tests/reader_tests/_li_test_utils.py Outdated Show resolved Hide resolved
satpy/tests/reader_tests/test_li_l2_nc.py Show resolved Hide resolved
Copy link
Member

@ameraner ameraner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@ameraner ameraner added enhancement code enhancements, features, improvements component:readers bug labels Dec 5, 2024
Copy link
Member

@pnuu pnuu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One suggestion inline.

Maybe we should start splitting the long test helpers to smaller chunks at some point?

I also restarted the one windows CI run.

@@ -158,11 +158,9 @@ def get_array_on_fci_grid(self, data_array: xr.DataArray):
data_2d = da.map_blocks(_np_add_at_wrapper, data_2d, (rows, cols), data_array,
dtype=data_array.dtype,
chunks=(LI_GRID_SHAPE[0], LI_GRID_SHAPE[1]))
data_2d = da.where(data_2d > 0, data_2d, np.nan)

data_2d = da.where(data_2d > 0, data_2d, np.nan).astype(np.float32)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would prevent one unnecessary upcasting:

Suggested change
data_2d = da.where(data_2d > 0, data_2d, np.nan).astype(np.float32)
data_2d = da.where(data_2d > 0, data_2d, np.float32(np.nan))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pnuu casting a nan who is a float32 is not enough to convert all the arrays into a float32 . For example if data_2d is an int32 the method where will convert it to a float64. To prevent it I have used the method astype(np.float32).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug component:readers enhancement code enhancements, features, improvements
Projects
Status: Ready for review
Development

Successfully merging this pull request may close these issues.

Unexpected floats when reading LI L2 LFL data
4 participants