IPIP-280: App Conventions for HTTP Gateway Detection

[markg85]

This should help other application developers to have an idea of what's needed to add IPFS support to their application.

cc @Stebalien and @autonome Conversations with mainly you two have led to this spec being implemented as is written here in ffmpeg.

[markg85]

Ping. What's still required for this document to get merged?
Do note that the document is called INTEGRATION.md because it's intended to be used by those "integrating" IPFS support in their applications.

Don't confuse this with the document the add the gateway file in the IPFS data folder. While this document "sort of refers" to it in terms of functionality, it doesn't spec - nor should it - how that file is created. That detail should be of no concern to those integrating IPFS support.

[pimterry]

Hi @markg85, two quick comments on this from some related discussion in ipfs/in-web-browsers#197:

• Could we extend this just slightly, to also specify gateway detection for browser JS environments? This is a common use case, which I think could easily closely match what you're suggesting here, by using a global window.IPFS_GATEWAY env var to allow browser and browser extensions to inject gateway configuration into JS apps. Similarly, those apps should use that configuration if available, or use the fallback gateway if not.

• This spec covers gateways only, but that's not the only way people may want to integrate IPFS into their applications. There's many interesting use cases where 3rd-party applications might want to interact with the network more generally, by embedding a full IPFS node. When doing so, it would be useful to provide very similar mechanisms to detect a user's preferred network configuration (particularly for web applications - see As a user, there is no mechanism to control the IPFS networking of a webapp I use in-web-browsers#197 for lots more context). Of course I don't want to derail this PR, which can happily proceed ignoring all that, but I think this would fit well as an extension to this 'Integration' guide later on. Do you think that makes sense? How about if I open up a PR against your branch here to start discussion on that too, but independently?

[lidel]

@markg85 time to revisit this, now that we have IPIP process. some pointers below :)

[markg85]

Hi @markg85, two quick comments on this from some related discussion in ipfs/in-web-browsers#197:

• Could we extend this just slightly, to also specify gateway detection for browser JS environments? This is a common use case, which I think could easily closely match what you're suggesting here, by using a global window.IPFS_GATEWAY env var to allow browser and browser extensions to inject gateway configuration into JS apps. Similarly, those apps should use that configuration if available, or use the fallback gateway if not.

I get the mindset here and it would help a lot! Not just with apps using IPFS but also with js libraries (think for example video.js) where we'd like to have IPFS integration in at some point in time too. Having a window.IPFS_GATEWAY would surely help!

But.. how?

When IPFS Companion is used, injecting a variable is doable.
But for IPFS integration in brave, that would be a code change on the brave side.

The downside is another point of metric collection to profile people. And for that reason alone i don't think this variable would be accepted. Even if we would all like it and add it to the spec, the browser security minded folks would probably have an issue with it.

@lidel do you have an opinion on this?

• This spec covers gateways only, but that's not the only way people may want to integrate IPFS into their applications. There's many interesting use cases where 3rd-party applications might want to interact with the network more generally, by embedding a full IPFS node. When doing so, it would be useful to provide very similar mechanisms to detect a user's preferred network configuration (particularly for web applications - see As a user, there is no mechanism to control the IPFS networking of a webapp I use in-web-browsers#197 for lots more context). Of course I don't want to derail this PR, which can happily proceed ignoring all that, but I think this would fit well as an extension to this 'Integration' guide later on. Do you think that makes sense? How about if I open up a PR against your branch here to start discussion on that too, but independently?

Usually i'd say "yes, go for it!" but in this case, waiting is probably the smarter move. If you'd go for that you'd have a spec wip branch on top of my wip branch. That's a little too much wip. Wait till this spec lands (it's mostly there) before you consider making a spec on top of this one.

[markg85]

@markg85 time to revisit this, now that we have IPIP process. some pointers below :)

Sweet!
Will update it all in the next couple days. Thank you for all the pointers, much appreciated!

[lidel]

Quick feedback:

• make this Kubo-agnostic
• document configuration location on MS Windows and macOS
• ensure manually added entries are not removed by starting/stopping local node
• allow for multiple local gateways to co-exist, and not "steal" traffic

[dirkf]

For Windows, it's not clear that the Roaming config (%APPDATA%) is suitable. You may not want your gateway setting on machine A (eg, with no local proxy) to be synced to machine B (eg, with a local proxy that has set a global gateway file). Perhaps use %LOCALAPPDATA%, or let it take precedence?

[dirkf]

Shouldn't the spec should say what text format is used? Is the filename in the file limited to ASCII, or otherwise is the file UTF-8, or in the encoding for the system locale, the system file encoding, etc?

[dirkf]

What happens if a gateway adds its URL to the global gateways and crashes without removing it (say, power loss)? Should there a clean-up system and/or some way to separate admin-specified defaults at the end of the file from dynamic entries at the start of the file?

[lidel]

@markg85 I've rebased and split IPIP into spec and IPIP memo, integrated feedback from @dirkf and others, and turned this into a spec we would be sending to applications to simplify integrations like ones in ffmpeg, curl etc.

lmk if I've removed something that is still relevant, or if you feel we should include something new.

[lidel]

@markg85 I'm not feeling strongly about having OS-specific paths or ~/.config and /etc: keep or remove?
afaik nothing implements these atm, but does not hurt to have them is spec to future-proof things.

[markg85]

I'm not feeling strong about those either.
They don't hurt or complicate things. The only reason i'd lean towards keeping it is because they clarify the windows part, mac/linux is effectively handled by points 1-4 as well.

So if it doesn't hurt, keep it. Else remove it.

[lidel]

@markg85 I wrote this bit naively, and now I realized that having more than one URL will cause issues in curl (code).

Any concerns if we simplify the spec and state that only one URL is allowed in the gateway file?

[markg85]

Ohh ouch!

For this spec. I'd keep it at a first line requirement but not limited to 1 line. Aka, it's fine as-is.

Curl needs some attention to catch the multiline scenario regardless of this spec. As just any other data after the first line is gonna break it.

[lidel]

Ok, my understanding is this problem was solved in curl/curl@859e88f#diff-1fa338b6a8d4b9a9ffd2243527ac4951c2576cb32dc28440f36f4f59e60643e2R785, the gateway file can now have more than one line, curl ignores everything after the first one.

[markg85]

Yes!

I'm leaving it up to your digression if you want to follow the curl 8.4.0 approach (meaning the spec needs to be reworded to 1 line and 1 only) or the curl 8.5.0 approach (first line must be the gateway, rest is ignored).

I would go for the 8.5.0 approach (aka, the spec as-written is ok). Given the other IPFS fixes in 8.5.0, that release is certainly the one to recommend for IPFS usage. Also, it's about to be released on Dec. 6th 2023.