Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
interactive plotting with
lonboard
#67interactive plotting with
lonboard
#67Changes from 7 commits
302a97b
3606fb2
0851443
774cf3f
2ffbb5f
6348ce2
0c2dc44
41856c1
091a7bd
2ef3192
46411a4
05495b8
5149fa4
1ec8bf3
269f684
27a6193
f7ddf09
e2958b7
f96a64b
49b54fe
20fd427
8117e7d
0a2f577
71f612a
998e6fb
0d31059
5a71559
76ffe15
84938a1
6ca24de
6d9b78a
c891e12
File filter
Filter by extension
Conversations
Jump to
There are no files selected for viewing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It doesn't look like you have docs yet, but if you use sphinx or mkdocs (with mkdocstrings), lonboard implements intersphinx, so you should be able to set it up so the return value is a hyperlink to the lonboard docs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks for the hint. I'll probably use
sphinx
to set up the docs, so I'll definitely make use of that!There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In case you have interest, it would be possible to implement this conversion in a way that doesn't require GeoPandas (or pyarrow), by using raw numpy buffers and arro3, a more minimal implementation of Arrow in Python that Lonboard adds as a dependency in the next version.
This would also be faster to render as it doesn't have to go through GeoPandas (and shapely geometries).
But it would be a little more complex code here. I can give you some pointers if you want.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd definitely be interested.
Right now, what we're doing is ask the user to pre-compute cell vertices from the cell ids (which can have various forms for different DGGS, but most often are
int64
oruint64
arrays), then convert them intoshapely
polygons. This is one of the things that take the longest other than converting togeopandas
and constructing theSolidPolygonLayer
, so if we could instead extenddeck.gl
to directly support them (like it supports H3 ids), that would be best. However, I suppose that will either take a long time and require someone to actually do the work, especially since there's so many different kinds of DGGS (and I'm not confident in my TypeScript / JavaScript skills so I won't be able to work on this myself anytime soon).So as the next best step, should we construct
geoarrow
polygons from the cell vertex coordinates, possibly usingarro3
? If you have any examples I'd be happy to try figuring out how to do that.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
At the end of the day, we need to create vertex information in Lonboard/deck.gl for rendering. I believe that even deck.gl's H3 layer does not render the h3 cells on the GPU from an ID, but rather uses those IDs on the CPU in JS to create the vertex geometry.
For now, I think it's simplest to convert cell ids to Polygon geometries in Python, and then we can use the
SolidPolygonLayer
/PolygonLayer
directly.A GeoArrow Polygon array is just:
Here's a quick code example that might help illustrate:
(Note that you need to use latest main of
arro3
, because I just implemented support for using numpy-backed memory for Rust Arrow arrays kylebarron/arro3#204, kylebarron/arro3#208)(Note that this simple implementation probably breaks over the antimeridian.)
h3-py is not itself vectorized, so converting from h3 cell ids to polygon vertices is quite slow. In the longer term, I'd like to update https://github.com/nmandery/h3ronpy to have a fast way to convert from h3 cell ids to GeoArrow polygon array (or MultiPolygonArray in the case of cells that cross the antimeridian)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That looks great @kylebarron !
Interesting. Do you know if deck.gl has some clever optimizations for plotting H3/S2 like, e.g., compute on the fly the cell polygon geometries and/or aggregating data (parent cells), depending on the current zoom level / display extent? This might still have some benefits for plotting very large datasets (common use case) interactively. Or maybe Lonboard allows doing such optimizations on the server side (widget callback)?
I guess this could really help better integration between arrow and xarray in general? 👀
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
h3ronpy
hash3ronpy.arrow.vector.cells_to_wkb_polygons
which would require you to decode WKB, so not sure if that's better.In any case, I've translated my example from #11 to use your example (while borrowing code from
lonboard.viz
to construct the arrow table and construct aSolidPolygonLayer
):(this also doesn't fully resolve the dateline issue)
which takes ~19 s to complete, plus about 10s to render. The version that converts to geopandas takes ~54s plus another ~10s to render, so this is a huge improvement.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's cool! I should figure out how to use
pyinstrument
. Was it a one-line thing to create that?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it has different modes for how you can run it (see the user guide). I put the code above in a python script, then ran it with
python -m pyinstrument lonboard_example.py
, but you could also use the context manager:or the jupyter / ipython cell magic
%%pyinstrument
(load first with%load_ext pyinstrument
)There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the ones beneath
SolidPolygonLayer
? Here you go:There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the most recent commits integrate the code we talked about into
xdggs
. If you have time to look at I'd appreciate another review, but either way thanks for all the help, this is already so much faster than the code I had before.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FWIW the next version of
geoarrow.rust.core
will have constructor functions: geoarrow/geoarrow-rs#810. And so you can pass the arrays directly intogeoarrow.rust.core.polygons
. Using those functions would ensure that the arrays generated are fully spec-compatible.(As it is, you're generating data whose physical buffers are the same as geoarrow arrays, but you're missing some metadata like the correct field names for nested arrays and the correct nullability flags for nested arrays. Lonboard allows that input but it's not 100% to the geoarrow spec, especially if we enforce it in issues like geoarrow/geoarrow#64)