-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Functions for reading metrics remotely from Azure #16
Conversation
42f5cba
to
bc0cfd2
Compare
Some benchmarks Without geo filteringQuery plan
With geo filteringQuery plan
This is a bit weird and I am wondering if the issue is the large header for this file (which has about 7000 columns). Perhaps revisit this once we have the data split in to multiple smaller parquet files. |
5000122
to
f875f9e
Compare
Co-authored-by: Jonathan Yong <[email protected]>
Broadly: I'm wondering about use cases. Is there a situation where we want to get the same metric for different geometries (e.g. maybe different countries)? In that case would it be fair to say that it is the user's responsibility to call |
This PR adds a function that takes a list of MetricRequests and fetches the data from cloud storage over http range requests in an efficient manner.
This generates the following results
There is also a way to filter by GEOIDs as we do so
which gives the result
TODO
The looks like the geo filtering version of the code is slower than the non geo filtering version. This is a bit counter intuitive so I want to properly benchmark it to see if that's true and try and figure out why.Opened ticket to follow up on this Explore issues with performance when using geo filtering with metrics #17