feat(gateway): more explicit IPFSBackend and no multi-range

[aschmahmann]

Also, switches to only handling single range requests rather than multiples.

• Merge conformance tests updates: refactor: multi range request considers servers that do not return multi range gateway-conformance#113
• Release conformance tests updates
• Run new conformance tests
• Kubo PR for removing outdated sharness: chore: remove sharness tests ported to conformance testing kubo#9999
• Kubo PR feat(gateway): update gateway api, no multi-range support kubo#10024

[codecov]

Codecov Report

Merging #369 (dcd7a19) into main (76f8e3f) will decrease coverage by 0.69%.
The diff coverage is 34.42%.

@@            Coverage Diff             @@
##             main     #369      +/-   ##
==========================================
- Coverage   66.44%   65.75%   -0.69%     
==========================================
  Files         204      208       +4     
  Lines       24621    25099     +478     
==========================================
+ Hits        16360    16505     +145     
- Misses       6838     7140     +302     
- Partials     1423     1454      +31     

Files	Coverage Δ
gateway/metrics.go	79.23% <0.00%> (ø)
gateway/handler_unixfs_file.go	73.01% <86.66%> (+8.73%)	⬆️
gateway/handler.go	78.01% <28.57%> (-0.91%)	⬇️
gateway/handler_unixfs__redirects.go	54.34% <61.53%> (+0.58%)	⬆️
gateway/handler_block.go	69.76% <22.22%> (-14.02%)	⬇️
gateway/handler_codec.go	60.95% <65.00%> (-1.24%)	⬇️
gateway/handler_unixfs_dir.go	60.25% <44.82%> (-4.03%)	⬇️
gateway/gateway.go	81.88% <40.00%> (-11.87%)	⬇️
gateway/handler_defaults.go	31.76% <38.88%> (-5.89%)	⬇️
gateway/blocks_backend.go	42.05% <24.44%> (-1.69%)	⬇️
... and 1 more

... and 14 files with indirect coverage changes

[hacdias]

@aschmahmann can we not support multiple ranges? Isn't that part of the specs? Also note that I'll be working on IPIP-412 soon, sow e might get some conflicts.

[BigLep]

2023-07-11 maintainer conversation: we shouldn't stress about multiple ranges. We have a metric for this against ipfs.io and haven't found any actual usage. This conformance test for multiple ranges can be moved to Kubo.
@aschmahmann will flip it out of draft when ready for review.

[hacdias]

This conformance test for multiple ranges can be moved to Kubo.

@BigLep @aschmahmann I don't understand how that helps. If we move the test to Kubo, it will still fail since we are removing the functionality on this PR and that functionality will simply no longer be available.

[aschmahmann]

If we move the test to Kubo, it will still fail since we are removing the functionality on this PR and that functionality will simply no longer be available.

Correct. IIUC the right thing to do here is:

1. Have the conformance tests test the HTTP guarantees around multiple ranges (e.g. that the server can send back the ranges, the first range, or none as long as it's clear from the response headers)
2. If we end up having this functionality in just one implementation/consumer of boxo (e.g. kubo) then we test there

We can add back in the multiple range functionality, but IIUC from @lidel basically no one is using this so it's not worth the hassle at the moment.

[hacdias]

Working on updating the tests here: ipfs/gateway-conformance#113

[hacdias]

@aschmahmann @lidel this PR is blocked until ipfs/gateway-conformance#113 is merged and released (ready for review), so that we can have the tests passing here and/or in Kubo's PR.

[lidel]

(submitting first-pass at the end of day)

[BigLep]

2023-08-22 conversation: @lidel will review and figure out appropriate testing story (Boxo unit tests vs. conformance tests)

[BigLep]

2023-08-24 conversation:
We are going to lean on conformance tests. @lidel is going to create a gateway-conformance issues for this.
It's a lesser evil to merge without the conformance tests than keep this open and let it diverge.
@aschmahmann will add some Boxo unit tests and flag the changed lines of potential concerns.
@aschmahmann will make sure we get the merge decision
(There is also some dance for keeping CI green with sharness.)

[BigLep]

@aschmahmann : is this on track to land this week?

[BigLep]

2023-09-12 conversation:

• Do a gateway-conformance release as soon as possible: feat: range request helpers gateway-conformance#162 lands and passes.
• To get green CI
• Conformance release
• Bubble up to Boxo
• Bubble this PR into Kubo
• Run this PR against Kubo version
• Once Boxo CI passes
• Switch to target Kubo master
• Merge this PR
• Bubble up Boxo commit to Kubo PR
• Merge Kubo PR
• Rerun Boxo CI

[hacdias]

CI is green in Kubo. As per what was agreed, I'm merging this once CI finishes here and bubbling it up.