feat: remote backend #117
feat: remote backend #117
Conversation
hacdias
force-pushed
the
remote-backend
branch
2 times, most recently
from
f7a2098
to
8d7348a
Compare
hacdias
requested review from
lidel and
aschmahmann
and removed request for
lidel
hacdias
force-pushed
the
remote-backend
branch
2 times, most recently
from
f5a8b9c
to
f1ea37f
Compare
hacdias
force-pushed
the
peering-cache
branch
2 times, most recently
from
17cb3e8
to
a5aa71e
Compare
There was a problem hiding this comment.
Thanks, nearly there :)
We need to streamline configuration a bit – fetch my changes, I've tweaked some docs, but need your help with comments inline.
Second ask is to extend gateway-conformance.yml Github Actions workflow in this repo to ensure block/car backendns behave the same way as running libp2p and bitswap.
In addition to current test (with lib2p), it should run separate backend tests similar to ones from https://github.com/ipfs/bifrost-gateway/blob/0bb1a904cc4af6c0b5e3398a67f320c1eec383a1/.github/workflows/gateway-conformance.yml#L14
main.go
Outdated
| bitswap := cctx.Bool("bitswap") | ||
| dhtRouting := DHTRouting(cctx.String("dht-routing")) | ||
| seedPeering := cctx.Bool("seed-peering") | ||
| noLibp2p := !bitswap && dhtRouting == DHTOff && !seedPeering |
There was a problem hiding this comment.
💭 noLibp2p setup feels overly complicated, and baloons the surface or permutations we need to test.
In practice either user will run their own libp2p node for dht+bitswap, or they will run no libp2p at all, and delegate both retrieval and routing to remote HTTP endpoints.
@hacdias Could you tweak it to have an explicit, global RAINBOW_LIBP2P=true|false (with true being default) that controls when SetupNoLibp2p is used instead of Setup[WithLibp2p]?
Then, inside SetupNoLibp2p just check if bitswap, dht-routing, seed-peering, libp2p flags are set to [expected values], and error otherwise.
We want end user's configuration to be simple:
RAINBOW_LIBP2P=false RAINBOW_REMOTE_BACKENDS=http://trustless-gateway.link ./rainbow
There was a problem hiding this comment.
Then, inside
SetupNoLibp2pjust check ifbitswap,dht-routing,seed-peering,libp2pflags are set totrue, and error otherwise.
Shouldn't they be false and dht-routing=off? If we want to check if they haven't been set, we need to do that before calling SetupNoLibp2p.
There was a problem hiding this comment.
I have pushed a commit with a global flag
hacdias
force-pushed
the
remote-backend
branch
4 times, most recently
from
ad48392
to
5271857
Compare
lidel
force-pushed
the
remote-backend
branch
6 times, most recently
from
f4d6632
to
e525f8b
Compare
There was a problem hiding this comment.
Thank you once again for salvaging this work @hacdias ❤️
I've pushed some test/docs, and simplified UX a bit, but kept both LIBP2P and BITSWAP flags, as they will be useful for testing when we start experimenting with HTTP retrieval.
Please take a look, and if no concerns, feel free to merge.
(we want to ship this as v1.3.0, but probably need to land #138 first)
| // as a convenience to the end user, and to reduce confusion | ||
| // libp2p is disabled when remote backends are defined | ||
| remoteBackends := cctx.StringSlice("remote-backends") | ||
| if len(remoteBackends) > 0 { | ||
| fmt.Printf("RAINBOW_REMOTE_BACKENDS set, forcing RAINBOW_LIBP2P=false\n") | ||
| libp2p = false | ||
| bitswap = false | ||
| dhtRouting = DHTOff | ||
| } |
There was a problem hiding this comment.
ℹ️ ended up with this, to ensure we are not running into unexpected edge cases that we missed in test coverage, or may have in future after someone refactors things.
TLDR: for end user, logic is simple: setting a remote backend effectively disables libp2p and related features, and only HTTP backend and routers remain.
End user UX is:
$ RAINBOW_REMOTE_BACKENDS=http://127.0.0.1:8080 ./rainbow
Starting rainbow dev-build
RAINBOW_REMOTE_BACKENDS set, forcing RAINBOW_LIBP2P=false
These are only applied to libp2p nodes, not the entire rainbow.
Block and CAR backends have significate difference in code paths and abstractions. We need to run conformance against them separately to ensure both code paths behave the same way.
Setting RAINBOW_REMOTE_BACKENDS should be enough to turn off libp2p and delegate everything to HTTP endpoints.
|
Merged ipfs/github-mgmt#205 and removed old conformance as required (we can add new one after this PR is mergeD) @hacdias you should be able to merge now. If so, mind shipping as v1.3? |
On top of #111. Closes #88. This PR adds support for remote backends (as described in #88). I made it such that it is the most flexible as I could. It should in scenarios where Libp2p is completely disabled (no bitswap, no dht), but also if libp2p needs to be enabled for things such as peer routing for peering and such.