You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As part of the effort to ameliorate memory issues during alert fetching, parsing, and caching (as discussed in this working document), we need to refactor the alert geometry loop.
The issue is that geometry fetching can, in the worse cases, return megabytes of point data. These points are then put into an in-memory union function in order to compute the encompassing geometry of all constituent (county or zone) geometries.
This ticket calls for limiting the number of requested geometries we process at a time when fetching county or zone information. The size of each chunk should be configurable, but as a first pass we want to set it to 5 at the most.
Here is a possible algo:
Make a RESULT_LIST to hold the computed chunks
Fetch the next CHUNK_SIZE county/zone geometries from the database
Union the fetched chunks into a single geometry
Append computed union to RESULT_LIST
If there are county/zones remaining, go back to step 2
If there are no more county/zones remaining, union the geometries in the RESULT_LIST
That should, when all is said and done, get you one single union for all constituent county/zone geometries without having to do one single and potentially memory intensive fetch and union computation.
Acceptance criteria
Code has been refactored to only fetch and process X (5) number of county/zone geometries at a time
Test the code with the worst case scenario locally in a low-memory docker instance, and observe memory consumption
The text was updated successfully, but these errors were encountered:
Description
As part of the effort to ameliorate memory issues during alert fetching, parsing, and caching (as discussed in this working document), we need to refactor the alert geometry loop.
You can find the relevant code here.
The issue is that geometry fetching can, in the worse cases, return megabytes of point data. These points are then put into an in-memory
union
function in order to compute the encompassing geometry of all constituent (county or zone) geometries.This ticket calls for limiting the number of requested geometries we process at a time when fetching county or zone information. The size of each chunk should be configurable, but as a first pass we want to set it to 5 at the most.
Here is a possible algo:
That should, when all is said and done, get you one single union for all constituent county/zone geometries without having to do one single and potentially memory intensive fetch and union computation.
Acceptance criteria
The text was updated successfully, but these errors were encountered: