You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Freeway doesn't do super well in general reading a ton of CAR shards. This will be better moving forward as we're able to store larger shards, but it's something we need to figure out for historical data (where performance here has the potential to get even worse as we move to Saturn, etc.).
The key use case to focus on here is when users look to read individual files within a large directory split across many CARs. Right now we have to read all the shards to get to where the file is in the directory. This definitely has the potential to improve (vs. rendering the root directory structure, which always needs to grab all the CAR shards).
One idea @vasco-santos had was to keep an index of the location of the individual files after the first, slower read, so every subsequent read is more performant. If we do choose to have such a data structure, it'd be good if it was implemented in a similar way to DUDEWHERE and SATNAV so we can utilize this in any future models for Saturn, etc. (e.g., public claim-based model).
The text was updated successfully, but these errors were encountered:
related, I think we should use the upload/add step where a user tells us that "this content root is in these CAR CIDs" as an opportunity to make a single aggregated index of all the individual CAR indexes as a single file / single sub-request fetch which would then let us find out what cars we need to handle any path request.
Freeway doesn't do super well in general reading a ton of CAR shards. This will be better moving forward as we're able to store larger shards, but it's something we need to figure out for historical data (where performance here has the potential to get even worse as we move to Saturn, etc.).
The key use case to focus on here is when users look to read individual files within a large directory split across many CARs. Right now we have to read all the shards to get to where the file is in the directory. This definitely has the potential to improve (vs. rendering the root directory structure, which always needs to grab all the CAR shards).
One idea @vasco-santos had was to keep an index of the location of the individual files after the first, slower read, so every subsequent read is more performant. If we do choose to have such a data structure, it'd be good if it was implemented in a similar way to DUDEWHERE and SATNAV so we can utilize this in any future models for Saturn, etc. (e.g., public claim-based model).
The text was updated successfully, but these errors were encountered: