IPIP-462: Ipfs-Path-Affinity on Gateways #462
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
| --- | ||
| title: "IPIP-0462: Ipfs-Path-Affinity on Gateways" | ||
| date: 2024-02-16 | ||
| ipip: proposal | ||
| editors: | ||
| - name: Marcin Rataj | ||
| github: lidel | ||
| url: https://lidel.org/ | ||
| affiliation: | ||
| name: IP Shipyard | ||
| url: https://ipshipyard.com | ||
| relatedIssues: | ||
| - https://github.com/ipfs/kubo/issues/10251 | ||
| - https://github.com/ipfs/kubo/issues/8676 | ||
| order: 462 | ||
| tags: ['ipips'] | ||
| --- | ||
|
|
||
| ## Summary | ||
|
|
||
| This IPIP adds gateway support for optional `Ipfs-Path-Affinity` HTTP request header. | ||
|
|
||
| ## Motivation | ||
|
|
||
| Endpoints that implement :cite[trustless-gateway] may receive requests for a | ||
| single block, or a CAR request sub-DAG of a biger tree. | ||
|
|
||
| Not every CID is announced today, some providers limit announcements to | ||
| top-level root CIDs due to time and cost. | ||
|
|
||
lidel marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| What this mean for ecosystem? It should adapt. Over time, both clients and | ||
lidel marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| servers should leverage the concept of "affinity". | ||
|
|
||
| ## Detailed design | ||
|
There was a problem hiding this comment. I really feel like this is missing some guidance on what the values should or can be. With existing wording, it's too vague and could result in significant client pain of having to implement different values for different providers. Also: we should really speak to the how of implementing this server on the client and server sides: at least some best practices. E.g. how are we going to implement this in There was a problem hiding this comment. Clarified format in https://en-wikipedia--on--ipfs-org.ipns.inbrowser.dev/wiki/Books fetching necessary blocks from https://trustless-gateway.link it would have in every |
||
|
|
||
| Introduce `Ipfs-Path-Affinity` HTTP request header to allow HTTP client to | ||
| inform gateway about the context of block/CAR request. | ||
|
Comment on lines
+46
to
+47
There was a problem hiding this comment. What is the format of the data that goes here? Is it There was a problem hiding this comment. IMO, it should be There was a problem hiding this comment. I would much prefer an ipfs:// protocol instead of pathing, but i guess whatever the users is likely to have is better ux There was a problem hiding this comment. Went with content path, as we already talk about them all over the specs. NOTE: because UnixFS can have whitespace, Clarified format in |
||
|
|
||
| Client asking gateway for a block SHOULD provide a hint about the DAG the block | ||
| belongs to, if such information is available. | ||
|
Comment on lines
+49
to
+50
There was a problem hiding this comment. Is it only a strict IPLD DAG where we'd recommend this? It seems like you could plausibly do this for a set of related data that aren't explicitly linked via IPLD (e.g. a website that has HTML that loads jpegs from within the same or a different root DAG). There was a problem hiding this comment. This reminds me of some chats we've had about it being likely that a provider of "bafyFoo1" would likely also have "bafyFoo2" There was a problem hiding this comment. The way field format is specified, these can be arbitrary content paths. It is up to the client to provide a meaningful hint. Most of the time it will be the content path the client tries to load, but it could also be |
||
|
|
||
| Gateway being unable to find providers for internal block should be | ||
lidel marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| able to leverage affinity information sent by client and use CIDs of parent | ||
| path segments as additional content routing lookup hints. | ||
|
|
||
| ## Design rationale | ||
|
|
||
| ### User benefit | ||
|
|
||
| When supported by both client and server: | ||
|
|
||
| - Light clients are able to use trustless HTTP gateway endpoints more | ||
| efficiently, resume downloads faster. | ||
| - Gateway operators are able to leverage the hint and save resources related to | ||
| provider lookup. | ||
| - Content providers are able to implement smarter announcement mechanisms, | ||
|
There was a problem hiding this comment. What exactly do you mean by "smarter announcement mechanisms"? There was a problem hiding this comment. "roots" isn't really a word here. Root of what? At every layer in the DAG going up you could slice off the top of the tree and declare a new root. Every piece of content that needs to be independently addressable should be advertised. See https://github.com/ipfs/specs/pull/462/files#r1492996318 So at the very least if you make a block-request for the middle of a tar.gz file (where no part of the file really needs to be addressed on its own) you should be able to find it even if the provider has only advertised the root of the file. As mentioned in the linked comment I do think we need to be careful not to mislead people though. There was a problem hiding this comment.
Yes, that makes sense in theory, but if we go to the
Here, the only independently addressable content is the tar.gz file and for range block-requests in the middle, you'd need to pass the affinity header to be able to fetch that. If we're on the same page thus far, what would be smarter announcement mechanisms/strategies? Is the idea for those to be codec-aware in the sense that you could tell the node to advertise all UnixFS Files and Directories? There was a problem hiding this comment. @2color we have some ideas for smarter announcement mechanisms/strategies in ipfs/kubo#8676 and more actionable ipfs/kubo#10365 (which has wip implementation). ps. we have a concept of There was a problem hiding this comment. It sounds like "smarter" in this case means "frugal but smart" in the sense that it involves potentially less announcements that are effective enough for routing to work. There was a problem hiding this comment. Exactly, there are "frugal" things we could do on both client and server to announce less, but be as efficient at retrieval in real world usage patterns (website browsing, video streaming, download resume or parallel downloads etc). |
||
| without worrying that internal blocks are not announced. | ||
lidel marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| ### Compatibility | ||
|
|
||
| This is an optional HTTP header which makes it backward-compatible with | ||
| existing ecosystem of HTTP clients and IPGS Gateways. | ||
lidel marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| ### Security | ||
|
|
||
| The client is in control when the affinity information is sent in the header, | ||
| and implementation SHOULD allow end user to disable it in context where parent | ||
lidel marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| content path information is considered sensitive information. | ||
|
|
||
| ### Alternatives | ||
|
|
||
| N/A | ||
|
There was a problem hiding this comment. Might be worth explaining alternatives here like:
There was a problem hiding this comment. Thanks. Added answer for the first one in 61518c6 |
||
|
|
||
| ## Test fixtures | ||
|
|
||
| N/A | ||
|
|
||
| ### Copyright | ||
|
|
||
| Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Might be worth adding a reminder here that every piece of data that is supposed to be able to be accessed independently should be advertised.
Want to remind people to not walk themselves into silly problems like having data only accessible when passing the affinity header and then being confused when they can't get the data when it's included in another DAG. (e.g. I have the IPFS logo inside the DAG for docs.ipfs.tech but not advertised, but then someone uses the CID for the logo on explore.ipld.io but can't get the data because the affinity is wrong).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reworded Motivation section in de0b231 a bit, but I think we need to have a page about advertisement and content routing on https://docs.ipfs.tech rather than specs.
(loose digression, just to clarify what I am optimizing fix for)
While I get what the logo example was supposed to illustrate, it won't be a real world scenario: even when only top dir is announced, and no files, whoever authored explore.ipld.io would have the logo cached locally, and once the DAG is published, the affinity of /ipns/explore.ipld.io will be enough to read it, even if it is still not announced directly.
Saying "announce things you want to access independently" sounds good on paper but does not work for real world use cases such as HTTP Range requests for video streaming or resuming downloads of big files.
Streaming 32GiB video is an example of something where we can mostly agree we should not announce every internal CID, even tho one could say those internal CIDs are "accessed independently" when I paused and resumed the player (generated new range request for specific internal CIDs).
IPFS clients should leverage knowledge of affinity path, and ask providers of the root CID of the parent entity (in this case, the root of UnixFS DAG, or a directory it is in) for the data.