feat(gateway): TAR response format

[hacdias]

Summary

• Adds TAR response format to the HTTP gateway. Closes add tarfile/zipfile downloads to HTTP gateway #7746.
• Spec: IPIP-288: TAR Gateway Response Format specs#288

How to use it

HTTP clients can request TAR response by passing the ?format=tar URL
parameter, or setting Accept: application/x-tar HTTP header:

$ export DIR_CID=bafybeigccimv3zqm5g4jt363faybagywkvqbrismoquogimy7kvz2sj7sq
$ curl -H "Accept: application/x-tar" "http://127.0.0.1:8080/ipfs/$DIR_CID" > dir.tar
$ curl "http://127.0.0.1:8080/ipfs/$DIR_CID?format=tar" | tar xv
bafybeigccimv3zqm5g4jt363faybagywkvqbrismoquogimy7kvz2sj7sq
bafybeigccimv3zqm5g4jt363faybagywkvqbrismoquogimy7kvz2sj7sq/1 - Barrel - Part 1 - alt.txt
bafybeigccimv3zqm5g4jt363faybagywkvqbrismoquogimy7kvz2sj7sq/1 - Barrel - Part 1 - transcript.txt
bafybeigccimv3zqm5g4jt363faybagywkvqbrismoquogimy7kvz2sj7sq/1 - Barrel - Part 1.png

Tasks

• Update TarWriter to fail if there's UnixFS files pointing outside the root directory: fix: error when TAR has files outside of root go-ipfs-files#56 with tests.
• Create CAR fixtures
  • File points outside the root directory
  • File with relative path points somewhere inside the root directory
• Write tests
• Merge and release new version of go-ipfs-files: fix: error when TAR has files outside of root go-ipfs-files#56

Things to note:

• The linked PR (fix: error when TAR has files outside of root go-ipfs-files#56) also changes the behaviour of ipfs get --archive in the sense that if a UnixFS directory contains a path that points to outside of its root, it fails to build the TAR archive. This way we prevent possible malicious TAR archives in all fronts. The user is suggested to use CAR archives iff they want the raw data.
• Just like in the CAR streaming (feat(gateway): Block and CAR response formats #8758), we stream the TAR file. HTTP does not support a way to clearly indicate that there's an error when streaming. Currently, we use the same strategy of using a trailer header (X-Stream-Error) in order to indicate an error. Since browsers cannot detect this kind of errors (http.headers.Trailer - reading trailers not supported by Fetch API mdn/browser-compat-data#14703), we forcefully try to close the connection to generate a network error on the user side.

Closes #7746

[lidel]

Thank you for driving this @hacdias

Looks sensible, next steps:

• Update http-gateways/PATH_GATEWAY.md specs
  • Rationale: downloading UnixFS directory as TAR massively increases utility of gateways in use cases where CAR is not necessary, so this should be part of the spec
  • PR against my PR (Add HTTP Gateway Specs specs#283) for now
• needs sharness test that downloads a directory as TAR and saves it to disk (curl piped to tar command should be enough)

[hacdias]

@lidel thanks for the review. I added two tests (not sure if on the right file).

1. This only adds support for .tar.
2. Specs on IPIP-288: TAR Gateway Response Format specs#288
3. .tar.gz discussed on feat: gateway support for tar.gz #9034
4. Codecov is saying that the tests do not cover any of the code which I don't understand why.

[lidel]

(I have unfinished review of this pending and some local test code, re-assigning to myself, as some notes for implementers need to be bubbled up to IPIP)

[hacdias]

To test the security concern raised by @lidel in this comment regarding overwriting files by having custom relative paths inside a custom made UnixFS DAG, I... built my own DAG that tries to exploit that same issue.

CID: bafybeieq5yvoaqyrfvyiwqvi7hqhi54hyqhyatkn3pgbjxylpxdm5cj5t4 (CAR here)

└─dir
  │ file
  │ ../../../malicious-file		<- Would rewrite a file outside the root directory
└─../../malicious-dir			<- Would rewrite a directory outside the root directory
  │ file 

I just pushed a WIP version of a "Base Dir TAR Writer" which is essentially a copy of the TAR Writer in go-ipfs-files. The difference is that we always ensure that all files are under a certain base directory.

	if !strings.HasPrefix(fpath, w.baseDir) {
		fpath = strings.Replace(fpath, ".", "", -1)
		fpath = strings.Replace(fpath, "..", "", -1)
		fpath = path.Join(w.baseDir, fpath)
	}

It does what it is supposed to do (only tested locally, no testcase). Some things I'm still unhappy with:

1. We do not keep the intended file tree of the directory. Since all files that would fall outside of the base directory are now at the root of the base directory, the tree isn't maintained. A way to maintain the structure would be to traverse the entire file tree first, and "calculate" the correct root directory in order to preserve the entire file tree.
   • But maybe this would take more valuable processing time?
   • One of the reasons why I did not write a test case yet because it will change slightly depending on whether we want to preserve the whole tree or just truncate the odd files to the root directory.
2. It is just a copy paste of the original TAR Writer with 5 lines added. There may be an easier way to reuse the original code that I can't see right now.

Note that the original TAR Writer is also used in ipfs get --archive, which means that that command also suffers from the same security 'concern'. I assume that it is not a real concern since for a command line utility, we assume the user knows what they are downloading and its their responsibility. Let me know if I'm wrong here.

Thoughts? /cc @lidel

[BigLep]

I haven't been following the details closely, but I know we have had security issues with tar issues over the years. Can we make sure the security hardening is all in one place and shared across the stack (rather than copy and pasted)?

[hacdias]

I haven't been following the details closely, but I know we have had security issues with tar issues over the years. Can we make sure the security hardening is all in one place and shared across the stack (rather than copy and pasted)?

I definitely agree and I think it'd be better to just update this on go-ipfs-files. However, I also want to ensure that everyone agrees with the strategy. Right now, all files that would otherwise fall outside the root directory are forced to stay within the root directory. This does not keep the tree structure.

Yesterday during the colo @lidel mentioned that it could also be a great idea to just return an error and fail building the TAR. That is the behaviour of ipfs get, as ipfs get fetches a TAR and unpacks it . The unpacking fails if there are files that point outside the root directory. This error is triggered in tar-utils:

https://github.com/ipfs/tar-utils/blob/91e41e72cd61595a086c48862277c419a994d971/extractor.go#L189-L207

And also:

https://github.com/ipfs/tar-utils/blob/91e41e72cd61595a086c48862277c419a994d971/extractor.go#L156-L167

Instead of failing during the TAR unpacking, we can make it fail during the TAR creation, covering all cases where malicious TARs may be created. However, as @lidel mentioned yesterday, this may break someone else's use case (any idea?). But so would truncating the paths. Nowadays we always have CAR files where the user can always download the RAW, unmodified data if they please.

So two options:

1. Fail during creation.
2. Truncate file paths.

In either case, I will be updating the original TarWriter in https://github.com/ipfs/go-ipfs-files.

/cc @lidel

[lidel]

• I'd go with (1): an explicit error is better than silently mutating directory tree
• TAR processing should error when DAG has items with paths named ../ that go beyond the /ipfs/cid
  • error should note that relative UnixFS paths beyond /ipfs/cid root are not allowed for TAR, and suggest use of CAR instead
• create test fixtures for /ipfs/cid/sub/../dir (no error) and /ipfs/cid/../dir (should error) and include their CIDs in specs (IPIP-288: TAR Gateway Response Format specs#288)

[lidel]

Thank you @hacdias, let's include this in Kubo 0.17-rc1 (will merge after CI passes).

(pushed some cosmetic changes around Etag + added entry to changelog)