feat: add an aggregated velocity statistics table

[NoamGaash]

No description provided.

[OriHoch]

I tried to run this query manually for a single day

by adding WHERE date >= '2024-12-01' and date < '2024-12-02'

• to the HourlyAverages query - it takes 10 seconds and returns ~20,000 rows
• to the external query - it was running for more then 5 minutes so I stopped it

extrapolating to all the data we have - which currently is ~2 years the total number of rows will be 20,000 * 730 = 14,600,000

because it's a materialized view which stores all its rows and taking into account the indices, this will add significant size to the DB. Also, the load on the DB for running the materialized view refresh needs to be taken into account, we will need to run it periodically.

I think because the basic query takes 10 seconds per day, and assuming it won't run very often it's worth to just run the query itself, or maybe add a non-materizlied view.

[OriHoch]

^

[NoamGaash]

Thank you!
Ideally, I'd like to have some cache mechanism, because I want to expose an API endpoint to get a heatmap of velocity statistics, that I will call to from the frontent.
When considering 14 million rows, each row contains 7 values (8 bytes each I guess?) is about 56 bytes per row * 14,600,000 ~= 900MB. I can see how that's a lot.
The front-end will cache the responses, but it's still computationally heavy operation.
I'll read about non materialized views.
Thank you very much for the fast and educative feedback 🙏

[NoamGaash]

Hi @OriHoch ! I came out with this proposal to implement some kind of cache mechanism.

the last used column will store the last time a specific date was calculated. It will be used to remove old entries from the DB.

I made an implementation for the API and tested it locally
hasadna/open-bus-stride-api#44

[OriHoch]

1. Currently the API is strictly for SELECT and I want to keep it that way as introducing updates/inserts into it introduces a lot of complexity and risk I want to keep out
2. Caching mechanisms I would prefer not to implement on the DB as that also introduces risks and scale problems, if you want caching we can add a Redis server but I prefer if we can avoid it
3. The idea of adding a new table and populating it is good, but the way to do it is via the ETL system, you can add a task that runs daily, iterates over all the dates for which all the source data exists and for which no data is in the new table and inserts their data to this table. Most of the tasks in the open_bus_stride_etl do something like this. I would name this table a more generic name maybe something like siri_vehicle_locations_daily_stats so in the future we can add other statistics to this table