Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor interop alert geometry loop #2139

Open
2 tasks
eric-gade opened this issue Dec 3, 2024 · 0 comments
Open
2 tasks

Refactor interop alert geometry loop #2139

eric-gade opened this issue Dec 3, 2024 · 0 comments
Labels
Bug Something isn't working Eng

Comments

@eric-gade
Copy link
Collaborator

Description

As part of the effort to ameliorate memory issues during alert fetching, parsing, and caching (as discussed in this working document), we need to refactor the alert geometry loop.

You can find the relevant code here.

The issue is that geometry fetching can, in the worse cases, return megabytes of point data. These points are then put into an in-memory union function in order to compute the encompassing geometry of all constituent (county or zone) geometries.

This ticket calls for limiting the number of requested geometries we process at a time when fetching county or zone information. The size of each chunk should be configurable, but as a first pass we want to set it to 5 at the most.

Here is a possible algo:

  1. Make a RESULT_LIST to hold the computed chunks
  2. Fetch the next CHUNK_SIZE county/zone geometries from the database
  3. Union the fetched chunks into a single geometry
  4. Append computed union to RESULT_LIST
  5. If there are county/zones remaining, go back to step 2
  6. If there are no more county/zones remaining, union the geometries in the RESULT_LIST

That should, when all is said and done, get you one single union for all constituent county/zone geometries without having to do one single and potentially memory intensive fetch and union computation.

Acceptance criteria

  • Code has been refactored to only fetch and process X (5) number of county/zone geometries at a time
  • Test the code with the worst case scenario locally in a low-memory docker instance, and observe memory consumption
@eric-gade eric-gade added the Bug Something isn't working label Dec 3, 2024
@eric-gade eric-gade added the Eng label Dec 3, 2024
@eric-gade eric-gade moved this from Inbox to Prioritized Backlog in Roadmap & Sprint Board - Weather.gov 2.0 Dec 3, 2024
@eric-gade eric-gade moved this from Prioritized Backlog to Doing in Roadmap & Sprint Board - Weather.gov 2.0 Dec 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something isn't working Eng
Projects
Development

No branches or pull requests

1 participant