Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial implementation of EUMETSAT IASI-NG reader #2879

Open
wants to merge 52 commits into
base: main
Choose a base branch
from

Conversation

roche-emmanuel
Copy link
Contributor

@roche-emmanuel roche-emmanuel commented Aug 9, 2024

This PR introduces the initial implementation of the IASI-NG reader for Satpy.

The changes provided here are self-contained in three files:

  • satpy/readers/iasi_ng_l2_nc.py: The implementation file for the reader.
  • satpy/etc/readers/iasi_ng_l2_nc.yaml: The YAML configuration file for the reader.
  • satpy/tests/reader_tests/test_iasi_ng_l2_nc.py: The unit tests for the reader.

There are still a few points I need to address (this list may be extended progressively):

  • Check support for products with two-letter names (O3_ and CO_), as I don't think they will pass the current file pattern.
  • Confirm with the EUMETSAT team whether the processing logic needs to be adjusted (awaiting clarification on some points concerning the product file specs).
  • Extend the unit tests to perform more comprehensive checks on different products (currently only testing with TWV).
  • Clean the code and remove debug outputs (e.g., print(...) statements).
  • Clarify what needs to be done/added concerning the documentation of this reader.
  • Add my name to the AUTHORS.md file.

The idea here is to start this PR early in the development phase to get early feedback and review notes from you, ensuring we are on the right track for eventual PR acceptance.

If you have any feedback on this PR already, please let me know 😊! In the meantime, I will continue working on the points mentioned above and will provide additional details on the changes as they are made on the feature branch.

Thanks for your help 🙏!

@roche-emmanuel
Copy link
Contributor Author

Alright, so I see from the workflows above that there are some failures due to the formatting of the file, I should maybe start with this before moving forward with the remaining code changes 😉.

Copy link

codecov bot commented Aug 30, 2024

Codecov Report

Attention: Patch coverage is 98.42209% with 8 lines in your changes missing coverage. Please review.

Project coverage is 96.12%. Comparing base (0589f61) to head (3089b8d).
Report is 16 commits behind head on main.

Files with missing lines Patch % Lines
satpy/readers/iasi_ng_l2_nc.py 96.65% 7 Missing ⚠️
satpy/tests/reader_tests/test_iasi_ng_l2_nc.py 99.66% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2879      +/-   ##
==========================================
+ Coverage   96.10%   96.12%   +0.02%     
==========================================
  Files         377      379       +2     
  Lines       55162    55669     +507     
==========================================
+ Hits        53012    53511     +499     
- Misses       2150     2158       +8     
Flag Coverage Δ
behaviourtests 3.90% <0.00%> (-0.04%) ⬇️
unittests 96.21% <98.42%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@coveralls
Copy link

coveralls commented Aug 30, 2024

Pull Request Test Coverage Report for Build 12195051508

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 501 of 509 (98.43%) changed or added relevant lines in 2 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.02%) to 96.228%

Changes Missing Coverage Covered Lines Changed/Added Lines %
satpy/tests/reader_tests/test_iasi_ng_l2_nc.py 298 299 99.67%
satpy/readers/iasi_ng_l2_nc.py 203 210 96.67%
Totals Coverage Status
Change from base Build 12165642437: 0.02%
Covered Lines: 53757
Relevant Lines: 55864

💛 - Coveralls

@roche-emmanuel roche-emmanuel marked this pull request as ready for review October 9, 2024 06:25
@roche-emmanuel
Copy link
Contributor Author

Hi everyone,

We now think that this MR is ready for a review by the Satpy team, so please let me know if you have any feedback on it. Thanks 😉!

@roche-emmanuel
Copy link
Contributor Author

Bump :-)!

Hi @djhoese , @mraspaud ,

I haven't heard back from you concerning this MR. So I was just wondering if there was anything else I could do on my side to help getting this one into the "processing pipeline" ? Please let me know if you have anything to suggest ;-)

Thanks! 🙏

Copy link
Member

@mraspaud mraspaud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok I started reviewing this, and it looks sensible, but got distracted by all the comments, so I think we need to address this first.
So let’s start by reading this: https://santim0ren0.medium.com/clean-code-comments-f9eac4ada16d

In this PR I can see different types of comments:

  • comments on what the next line is: if the name of the variable or function is good enough, they are not needed
  • comments that explain what the following lines do: these lines should then be refactored into a function with a clear name
  • comments that complement the docstrings: these should be integrated to the said docstrings

I’m happy to answer any questions if you need help with this :)

satpy/readers/iasi_ng_l2_nc.py Outdated Show resolved Hide resolved
satpy/readers/iasi_ng_l2_nc.py Outdated Show resolved Hide resolved
satpy/readers/iasi_ng_l2_nc.py Outdated Show resolved Hide resolved
satpy/readers/iasi_ng_l2_nc.py Outdated Show resolved Hide resolved
satpy/readers/iasi_ng_l2_nc.py Outdated Show resolved Hide resolved
Comment on lines 160 to 173
# Prepare a description for this variable:
prefix, var_name = key.rsplit("/", 1)
dims = self.file_content[f"{key}/dimensions"]
dtype = self.file_content[f"{key}/dtype"]

desc = {
"location": key,
"prefix": prefix,
"var_name": var_name,
"shape": shape,
"dtype": f"{dtype}",
"dims": dims,
"attribs": {},
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that comment here surely means this code bit should be in its own function called eg prepare_description_for_variable?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @mraspaud ,

Thank you very much for this initial feedback 😊

I've read the article you mentioned, and even if I don't fully agree to this perspective on code comments I think I see your point, and will start working on the requested changes asap. I'll let you when all the fixes are integrated and you can have another look at this PR.

Thanks 🙏!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @mraspaud ,

I have now updated the content of the iasi_ng_l2_nc.py file removing those excessive comments (just keeping a couple of "dev notes" in the end).

Now, just to be sure before continuing: you would also expect me to do the same for the test_iasi_ng_l2_nc.py file, right :-)?

Also, I'm not sure if you would expect me to "resolve" each of the points reported above myself, or if you prefer to review the changes first and then resolve those points yourself when you juge the result acceptable (please just let me know if you think I should do it myself).

Thanks!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot, looks much better now!
yes it would be great if you could do the same in the test file.
Regarding the resolution of comments, I like it when the original commenter (in this case me) clicks the resolve button.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay 👌copy that: so I'm updating the test file now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @mraspaud ,

I have now also updated the test_iasi_ng_l2_nc.py file as discussed above. So please let me know when you find some time to give this PR another look, if you notice anything else that we should discuss/change.

Thanks in advance for your time on this 🙏!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants