Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Freeway performance reading content at /path where root directory CID is shared across many CARs #31

Open
dchoi27 opened this issue Mar 31, 2023 · 1 comment

Comments

@dchoi27
Copy link

dchoi27 commented Mar 31, 2023

Freeway doesn't do super well in general reading a ton of CAR shards. This will be better moving forward as we're able to store larger shards, but it's something we need to figure out for historical data (where performance here has the potential to get even worse as we move to Saturn, etc.).

The key use case to focus on here is when users look to read individual files within a large directory split across many CARs. Right now we have to read all the shards to get to where the file is in the directory. This definitely has the potential to improve (vs. rendering the root directory structure, which always needs to grab all the CAR shards).

One idea @vasco-santos had was to keep an index of the location of the individual files after the first, slower read, so every subsequent read is more performant. If we do choose to have such a data structure, it'd be good if it was implemented in a similar way to DUDEWHERE and SATNAV so we can utilize this in any future models for Saturn, etc. (e.g., public claim-based model).

@vasco-santos vasco-santos transferred this issue from storacha/w3infra Apr 10, 2023
@olizilla
Copy link
Contributor

related, I think we should use the upload/add step where a user tells us that "this content root is in these CAR CIDs" as an opportunity to make a single aggregated index of all the individual CAR indexes as a single file / single sub-request fetch which would then let us find out what cars we need to handle any path request.

see: storacha/w3link#49

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants