Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(gateway): JSON and CBOR response formats (IPIP-328) #9335

Merged
merged 39 commits into from
Dec 5, 2022
Merged

feat(gateway): JSON and CBOR response formats (IPIP-328) #9335

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
ac8f9b1
wip: play with dag-cbor and dag-json
hacdias Oct 6, 2022
3dce012
wip: add application/json and application/cbor
hacdias Oct 7, 2022
b557181
fix: go cases don't flow automatically :)
hacdias Oct 10, 2022
4104bb7
test: add some dag-json and dag-cbor tests
hacdias Oct 10, 2022
25893c5
test: improve names
hacdias Oct 10, 2022
e621e64
feat: allow json and cbor data types too
hacdias Oct 11, 2022
44946ed
refactor: avoid encoding things that are already on their right encoding
hacdias Oct 13, 2022
199ab42
fix: remove responseFormat from logging
hacdias Oct 13, 2022
89eb033
refactor: simplify serveCodec to use serveRawBlock iff data encoded i…
hacdias Oct 13, 2022
fb50869
tests: rename current tests to indicate they're unixfs only
hacdias Oct 13, 2022
fc31241
refactor: do not use serveRawBlock inside serveCodec bc headers and o…
hacdias Oct 13, 2022
55383cd
test: add test with pure json and cbor
hacdias Oct 17, 2022
a6d45c7
test: convert cbor <-> json
hacdias Oct 17, 2022
1986be1
test: path traversal and dag-pb output
hacdias Oct 19, 2022
9ef022e
fix: add more info about errors
hacdias Oct 20, 2022
cadc681
fix: add missing traversal
hacdias Oct 20, 2022
6331695
Merge branch 'master' into feat/8823
hacdias Nov 10, 2022
2c93672
fix: remove duplicate variable
hacdias Nov 10, 2022
462c71b
Merge branch 'master' into feat/8823
hacdias Nov 11, 2022
1e844c5
refactor: do not support traversal
hacdias Nov 11, 2022
bb98041
Update core/corehttp/gateway_handler_codec.go
hacdias Nov 15, 2022
53d5878
improve PR to match spec
hacdias Nov 16, 2022
4064f97
Merge branch 'master' into feat/8823
hacdias Nov 16, 2022
8c6a8da
feat: little web page
hacdias Nov 16, 2022
b5e5ff2
feat: update doc
hacdias Nov 16, 2022
8ca2a52
fix: Content-Disposition .json and .cbor
lidel Nov 23, 2022
b4dfa66
fix: inline disposition for JSON responses
lidel Nov 23, 2022
83913c7
refactor: return 501 for unsupported pathing
lidel Nov 23, 2022
ff55745
docs(cbor): improved info about codec
lidel Nov 23, 2022
52711d3
refactor: create template at assets/dag-index-html
lidel Nov 24, 2022
7e84856
fix(dag@gw): content type and cache headers
lidel Nov 25, 2022
3b89f20
Merge branch 'master' into feat/8823
hacdias Nov 28, 2022
162f435
add changelog info
hacdias Nov 28, 2022
12d0d7f
fix title
hacdias Nov 28, 2022
2d8ba78
rm wild block
hacdias Nov 30, 2022
b5874e7
Merge branch 'master' into feat/8823
hacdias Dec 5, 2022
0c08a76
Merge branch 'master' into feat/8823
hacdias Dec 5, 2022
f084f09
fix(dag-index-html): remove technical jargon
lidel Dec 5, 2022
32bcd41
Merge branch 'master' into feat/8823
lidel Dec 5, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions core/corehttp/gateway_handler_codec.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,15 +127,15 @@ func (i *gatewayHandler) serverCodecHTML(ctx context.Context, w http.ResponseWri
<pre>%s</pre>
<p>Please follow the following links to download the document in other formats:</p>
<ul>
<li><a href="?format=dag-json">DAG-JSON</a></li>
<li><a href="?format=dag-cbor">DAG-CBOR</a></li>
<li><a href="?format=raw">Raw</a></li>
<li><a href="?format=dag-json">Download in DAG-JSON</a> (<a href="https://ipld.io/specs/codecs/dag-json/spec/">format specification</a>)</li>
<li><a href="?format=dag-cbor">Download in DAG-CBOR</a> (<a href="https://ipld.io/specs/codecs/dag-cbor/spec/">format specification</a>)</li>
<li><a href="?format=raw">Download Raw Block</a></li>
</ul>
</body>
</html>
`, contentPath.String())

w.Write([]byte(body))
_, _ = w.Write([]byte(body))
}

func (i *gatewayHandler) serveCodecRaw(ctx context.Context, w http.ResponseWriter, r *http.Request, resolvedPath ipath.Resolved, contentPath ipath.Path, contentType string) {
Expand Down Expand Up @@ -204,7 +204,7 @@ func (i *gatewayHandler) serveCodecConverted(ctx context.Context, w http.Respons
w.Header().Set("Last-Modified", modtime.UTC().Format(http.TimeFormat))
}

w.Write(buf.Bytes())
_, _ = w.Write(buf.Bytes())
}

func setCodecContentDisposition(w http.ResponseWriter, r *http.Request, resolvedPath ipath.Resolved, contentType string) string {
Expand Down
10 changes: 10 additions & 0 deletions test/sharness/t0123-gateway-json-cbor.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,11 @@ test_expect_success "Add CARs for path traversal and DAG-PB representation tests
test_should_contain $DAG_PB_CID import_output
'

test_expect_success "GET DAG-JSON with Accept: text/html returns HTML" '
curl -sD - -H "Accept: text/html" "http://127.0.0.1:$GWAY_PORT/ipfs/$DAG_JSON_TRAVERSAL_CID" > curl_output 2>&1 &&
test_should_contain "Content-Type: text/html" curl_output
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ We were missing Cache-Control and Etag header tests – added them in 7e84856

  • unified the way we set headers for DAG- responses, and added missing ones for HTML + bunch of tests
  • we needed separate tests for HTML responses because it is a different code path AND caching needs to be different: generated HTML may change in future Kubo, so it is not immutable response, even when comes from /ipfs
    • the logic for Cache-Control/Etag for HTML response is similar to ones from dir-index-html responses produced by gateway_handler_unixfs_dir.go (see t0116-gateway-cache.sh for prior examples).

Future work:

  • potentially cool project would be to create a single view that handles both unixfs dir listings and non-unixfs dags, and shows more useful info about the DAG (I'm thinking "https://explore.ipld.io 2.0" :)) – not urgent, but somehting we would could start designing in cooperation with gui team.
  • if we evern change how dir-index-html responses behave, the analog changes should be applied to dag-index-html.

'

test_expect_success "GET DAG-JSON traversal returns 400 if there is path remainder" '
curl --head "http://127.0.0.1:$GWAY_PORT/ipfs/$DAG_JSON_TRAVERSAL_CID/foo?format=dag-json" > curl_output 2>&1 &&
test_should_contain "400 Bad Request" curl_output
Expand All @@ -154,6 +159,11 @@ test_expect_success "GET DAG-JSON traverses multiple links" '
test_cmp expected actual
'

test_expect_success "GET DAG-CBOR with Accept: text/html returns HTML" '
curl -sD - -H "Accept: text/html" "http://127.0.0.1:$GWAY_PORT/ipfs/$DAG_CBOR_TRAVERSAL_CID" > curl_output 2>&1 &&
test_should_contain "Content-Type: text/html" curl_output
'

test_expect_success "GET DAG-CBOR traversal returns 400 if there is path remainder" '
curl --head "http://127.0.0.1:$GWAY_PORT/ipfs/$DAG_CBOR_TRAVERSAL_CID/foo?format=dag-cbor" > curl_output 2>&1 &&
test_should_contain "400 Bad Request" curl_output
Expand Down