Area definition html representation for Jupyter notebooks #450

BENR0 · 2022-08-05T18:12:24Z

This adds _repr_html_ to the AreaDefinition class. In Jupyter notebooks therefore instead of a string a nicely formatted text is show together with a map overview (cartopy plot with borders and coastlines) as can be seen in the example below. The map by default is folded right now to save space (for the screenshot I unfolded it).

The way this is setup is heavily influenced by how xarray implements the _repr_html_. Basically I used the same directory/ file structure. I also copied some code (the html with the svg icons, the function to read the static data) which I did not attribute yet.

While functionally ready some considerations before this get merged:

Apart from some refactoring of the plotting function (removing unecessary code and comments) I was thinking about maybe moving the plotting function into the class?
Only the AreaDefinition has a html representation as of now. It would be nice if something similar could be done for SwathDefinition to make the look and feel more homogeneous.
Tests added
Tests passed
Passes git diff origin/main **/*py | flake8 --diff
Fully documented

codecov · 2022-08-05T18:20:04Z

Codecov Report

Attention: 9 lines in your changes are missing coverage. Please review.

Comparison is base (e82bf47) 94.06% compared to head (5191248) 94.23%.
Report is 2 commits behind head on main.

Files	Patch %	Lines
pyresample/_formatting_html.py	94.57%	7 Missing ⚠️
pyresample/area_config.py	92.30%	1 Missing ⚠️
pyresample/geometry.py	83.33%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #450      +/-   ##
==========================================
+ Coverage   94.06%   94.23%   +0.17%     
==========================================
  Files          85       87       +2     
  Lines       13250    13458     +208     
==========================================
+ Hits        12463    12682     +219     
+ Misses        787      776      -11

Flag	Coverage Δ
unittests	`94.23% <95.73%> (+0.17%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

coveralls · 2022-08-05T18:33:17Z

coverage: 93.822% (+0.2%) from 93.644%
when pulling 5191248 on BENR0:feat_html_repr
into e82bf47 on pytroll:main.

djhoese · 2022-08-05T18:50:40Z

You may want to try installing the pre-commit hooks. It looks like you've made the style checkers very angry 😉

Other than the style issues, what happens if matplotlib and/or cartopy are not installed? These are not hard requirements of pyresample so we should still show something useful if they are not installed.

djhoese

In addition to my previous comment, I think you need to make sure that all of these new non-python files are in the MANIFEST.in so they are included in the distributed package. Lastly, would it be possible to just include the .css and .html files as strings in the new formatting_html.py module? And can we prefix formatting_html.py with an underscore so users are tempted to look into it?

This is really cool. Thanks for putting in the time to get it working.

BENR0 · 2022-08-05T20:10:06Z

@djhoese Yeah I already saw it :-D. I think most of the complaints are in the plotting which I need to refactor a little anyway.

Good point about matplotlib/cartopy didn't think about that. But in that case I think it should just fall back to the normal string representation.

I will also Look into the other suggestions you made.

djhoese

A couple suggestions and questions, but otherwise coveralls is still saying the SwathDefinition branch of the formatting is not covered. That and the PNG output option. I see mention of swath definitions in your test so any idea why those branches aren't being covered?

djhoese · 2023-09-28T13:19:18Z

pyresample/_formatting_html.py

+try:
+    import cartopy
+    cart = True
+except ModuleNotFoundError:
+    cart = False


This is a personal preference, but you could avoid the extra boolean variable (here and for the xarray case) and do cartopy = None in the except case.

Do you want me to change it?

Sure. It's just me being nitpicky.

djhoese · 2023-09-28T13:20:31Z

pyresample/_formatting_html.py

+
+try:
+    import xarray as xr
+    from xarray.core.formatting_html import _obj_repr, datavar_section


Do we have to use a private function (_obj_repr)? How does it differ from repr(xarr_obj)?

Unfortunately yes. repr(xarr_obj) produces the string representation which is also used as a fallback/ in the non notebook case. I could just "copy" the function into pyresample and make it non private?

And your usage differs from the public array_repr and dataset_repr in that same formatting_html module in xarray (these functions use _obj_repr)?

Yes because I want to be able to customize the header and the displayed sections (I only want to display the data variables from the whole xarray.Dataset representation). Obviously that is a (personal) design choice which I am happy to talk about. I just tried to reuse as much as was already available but we could also implement something of our own.

Just to give you an idea of the differences (top: _obj_repr, bottom: dataset_repr):

Very interesting. Yeah I see what you mean. I was going to say maybe we copy that functionality here, but then noticed it is loading the static icon SVGs so that gets awkward too. I guess that function is small too but it feels weird to copy them.

@mraspaud @pnuu any opinions on this for what "feels" right from the above screenshots? The bottom has so much extra "cruft" for things that aren't used, but maybe that is OK since we are just completely depending on upstream xarray's "private" functions. Or... @BENR0 what about a PR to xarray (in the long term) that let's you exclude empty sections? That would still include the Attributes but maybe that's a good thing? I could see it being beneficial to users to see the extra metadata hanging around the swath definition.

Using a private function/method from another libraries feels wrong I have to say. I agree that the bottom is maybe too verbose, but having the dimension size explicitly here I think is nice actually.

BENR0 · 2023-10-05T06:04:23Z

A couple suggestions and questions, but otherwise coveralls is still saying the SwathDefinition branch of the formatting is not covered. That and the PNG output option. I see mention of swath definitions in your test so any idea why those branches aren't being covered?

I think the SwathDefinition tests just test the html code part and not the overview plot that's why that branch in the plotting function is not covered. I will try and add tests for the plotting function itself or update the test to also cover the SwathDefinition branch. The PNG output option could be removed I just left it there as a feature but it's not used in the html representation. Should I remove it then or add a test (even though I am not sure how write a check for that yet :-D).

djhoese · 2023-10-05T16:05:49Z

If the PNG stuff isn't used in the final output then I guess I'm ok with it being removed. Although...it might be nice to export an area definition to a PNG. A test wouldn't be too bad as long as you just checked that it was saving and maybe check that it isn't all black. I guess it is up to you what is best for the future. Or maybe @mraspaud has an opinion.

BENR0 · 2023-10-09T08:20:09Z

The problem with the testing of the plotting function is that I actually don't save the plot to disc but generate the string representation to include it in the html later on. I guess I could check if savefig was called which would be somehow equivalent to checking that something gets saved. I have no idea how to check if the output is all black or not.
/

djhoese · 2023-10-09T15:42:36Z

Given that we're basically depending on another program to output the PNG representation I am OK with just knowing that it isn't empty (0 size).

pyresample/_formatting_html.py

djhoese · 2023-10-09T15:45:30Z

Ah I see you added specific tests that check savefig.

pyresample/_formatting_html.py

djhoese · 2023-10-10T18:20:24Z

pyresample/_formatting_html.py

+@lru_cache(None)
+def _load_static_files():


Is there any reason for this not to be 1?

djhoese · 2023-10-10T18:21:46Z

pyresample/area_config.py

+
+    Args:
+        area_file (str): Path to area yaml file.
+
+    Returns:
+        str: rst list formatted string.


For future reference I think I'd like pyresample to migrate to type annotations and no type information in the docstring.

djhoese

A couple more comments, but otherwise I approve of this as long as @pnuu and @mraspaud are OK with that comment about importing private functions from xarray.

Co-authored-by: David Hoese <[email protected]>

…t_html_repr

mraspaud

LGTM. I think the import of the private function from xarray is indeed fragile, as there is no guaranty of that function's future. However, is seems to be something that is helpful, and I would thus suggest opening an issue in xarray to make this function public instead.

mraspaud

LGTM!

mraspaud · 2023-12-13T08:29:33Z

pre-commit.ci autofix

for more information, see https://pre-commit.ci

BENR0 · 2023-12-14T07:20:14Z

Opened an issue about the private _obj_repr function here: pydata/xarray#8551

BENR0 and others added 15 commits July 28, 2022 18:59

feat: basic implementation of html repr

924627c

fix: css wrapper width and plotting to inline svg

af2e9f5

fix: set_global error

e314e70

refactor: add docstring make header optional

c875136

add: extent description

1844e66

refactor: collapsible sections

1f8bc79

add: globe svg. Not working yet

9e59637

fix: indentation of details and formatting of properties list

e73913b

fix: globe icon

fb0b407

add: globe icon for map collapsible

af779f3

add: docstrings

225e86e

fix: list display

f2b0509

fix: plot showing a second time in jupyter lab

7baad8d

fix: comment

45fb3b3

Merge branch 'main' into feat_html_repr

fd1c358

BENR0 mentioned this pull request Aug 5, 2022

Automatic list with overviews of inlcuded area definitions for the documentation pytroll/satpy#2167

Merged

2 tasks

djhoese requested changes Aug 5, 2022

View reviewed changes

BENR0 and others added 3 commits August 8, 2022 10:17

reafactor: plotting function

b596a08

refactor: rename formatting_html

fab8e37

fix: static files missing in manifest

9d2fb5d

BENR0 mentioned this pull request Aug 11, 2022

Implements a html representation for the Scene pytroll/satpy#2171

Draft

Benjamin Rösner added 5 commits August 16, 2022 08:38

refactor: area plotting function to be able to plot SwathDefinitions

131afd0

feat: add initial html representation to SwathDefinition

9cdaf7e

refactor: add xarray style like view of lon/lat to swath definition

d0d8230

feat: add polygon plotting variant to swath plotting

affeda9

refactor: add satellite icon

42eabaa

Merge branch 'main' into feat_html_repr

55669d6

djhoese requested changes Sep 28, 2023

View reviewed changes

Benjamin Rösner added 2 commits October 9, 2023 13:15

refactor: plot_area_def function.

89b6663

test: add dedicated tests for plot_area_def.

99dc7ad

djhoese reviewed Oct 9, 2023

View reviewed changes

pyresample/_formatting_html.py Show resolved Hide resolved

djhoese reviewed Oct 10, 2023

View reviewed changes

pyresample/_formatting_html.py Outdated Show resolved Hide resolved

djhoese reviewed Oct 10, 2023

View reviewed changes

djhoese approved these changes Oct 10, 2023

View reviewed changes

BENR0 and others added 6 commits October 11, 2023 10:12

Update copyright year

2019450

Co-authored-by: David Hoese <[email protected]>

refactor: change docstrings in favour of type annotations.

f9a3e5b

refactor: cartopy import.

e99dabe

Merge branch 'main' into feat_html_repr

05dc307

refactor: make overall width smaller.

809057d

Merge branch 'feat_html_repr' of github.com:BENR0/pyresample into fea…

22b483a

…t_html_repr

mraspaud approved these changes Dec 12, 2023

View reviewed changes

mraspaud assigned BENR0 Dec 12, 2023

BENR0 added 3 commits December 12, 2023 15:26

fix: mock called with assertions.

7d6a9b8

fix: missed one.

d70f0ea

refactor: import future annotations.

bf32af3

mraspaud approved these changes Dec 13, 2023

View reviewed changes

[pre-commit.ci] auto fixes from pre-commit.com hooks

5191248

for more information, see https://pre-commit.ci

mraspaud merged commit b014586 into pytroll:main Dec 13, 2023
24 of 25 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Area definition html representation for Jupyter notebooks #450

Area definition html representation for Jupyter notebooks #450

BENR0 commented Aug 5, 2022 •

edited

Loading

codecov bot commented Aug 5, 2022 •

edited

Loading

coveralls commented Aug 5, 2022 •

edited

Loading

djhoese commented Aug 5, 2022

djhoese left a comment

BENR0 commented Aug 5, 2022 •

edited

Loading

djhoese left a comment

djhoese Sep 28, 2023

BENR0 Oct 5, 2023

djhoese Oct 5, 2023

djhoese Sep 28, 2023

BENR0 Oct 5, 2023

djhoese Oct 5, 2023

BENR0 Oct 9, 2023

djhoese Oct 9, 2023

mraspaud Oct 11, 2023

BENR0 commented Oct 5, 2023

djhoese commented Oct 5, 2023

BENR0 commented Oct 9, 2023

djhoese commented Oct 9, 2023

djhoese commented Oct 9, 2023

djhoese Oct 10, 2023

djhoese Oct 10, 2023

djhoese left a comment

mraspaud left a comment

mraspaud left a comment

mraspaud commented Dec 13, 2023

BENR0 commented Dec 14, 2023

		@lru_cache(None)
		def _load_static_files():

Area definition html representation for Jupyter notebooks #450

Area definition html representation for Jupyter notebooks #450

Conversation

BENR0 commented Aug 5, 2022 • edited Loading

codecov bot commented Aug 5, 2022 • edited Loading

Codecov Report

coveralls commented Aug 5, 2022 • edited Loading

djhoese commented Aug 5, 2022

djhoese left a comment

Choose a reason for hiding this comment

BENR0 commented Aug 5, 2022 • edited Loading

djhoese left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BENR0 commented Oct 5, 2023

djhoese commented Oct 5, 2023

BENR0 commented Oct 9, 2023

djhoese commented Oct 9, 2023

djhoese commented Oct 9, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

djhoese left a comment

Choose a reason for hiding this comment

mraspaud left a comment

Choose a reason for hiding this comment

mraspaud left a comment

Choose a reason for hiding this comment

mraspaud commented Dec 13, 2023

BENR0 commented Dec 14, 2023

BENR0 commented Aug 5, 2022 •

edited

Loading

codecov bot commented Aug 5, 2022 •

edited

Loading

coveralls commented Aug 5, 2022 •

edited

Loading

BENR0 commented Aug 5, 2022 •

edited

Loading