Support for IPFS

[james-pre]

@RangerMauve:

IPFS is a protocol that uses contrent-addressable file storage in a peer-to-peer network.
I think it would be useful to support it as a storage mechanism to reduce the amount of data that needs to actually be stored locally.

Instead of storing files on the local filesystem, one could store file metadata (names / created at) and have it point to an IPFS hash of that data, and when updating files, you would upload the new version to IPFS instead. This way one could load file data only when it's needed and free up user storage as well as improve the IPFS network by having more data stored between peers.

@jvilk:

Looks interesting! I'd accept a PR with an IPFS backend, although it looks like this request is somewhat premature as the IPFS JS library is still in "alpha" state.

@jvilk:

Also I have no idea how unit tests would work; is it possible to run an isolated IPFS instance for that purpose?

@RangerMauve:

@jvilk Now that IPFS has their circuit-relay stuff figured out, the next release should be a lot more stable and compatible across more transports.

With regards to testing, you can set up an IPFS node that doesn't have access to any other peers and has MDN disabled so it won't do peer discovery. I'd look at how they're doing unit tests to get a better picture of how they do it.

@jvilk:

Something that could be started up and stopped alongside the tests running in Travis CI and appveyor would be ideal, as I have no resources to run a dedicated testing server.

@RangerMauve:

You can start an IPFS node in either a node.js or browser context.

@neuronsupport:

I believe it is possible to run ipfs-js instance from node.js.

@willemneal:

I'm interested in working on this. I've been working with Orbit-db, which is built on top of IPFS's pubsub to allow CRDT append-only logs. Since each file/directory in IPFS is referenced by its hash, if any file changes you have a new reference.

Essentially Orbit-db creates a file that describes the log: its name, type, and who can write to it, which is controlled by providing a set of public keys that must correspond to a signature provided with each new entry into the log.

The pubsub feature that Orbit-db uses provide P2P WebSockets to connect all those interested in updates about a "topic." So when any new user arrives and is interested in an orbit-db, any connected user with the log can catch them up.

So what I propose is the following:

A filesystem (top level directory) is added to IPFS (it uses IndexedDB as to store the files, which are sharded) producing a hash reference. A corresponding orbit-db log is created and this becomes the permanent "reference" to the file system. The IPFS hash is then appended to the log. Then any subsequent change to the underlying file system results on it being added to IPFS producing a new hash and publishing the reference to the log.

Since IPFS uses a merkle tree for its directory structure, each file with the filesystem is referenced by its hash. So when a subscriber gets a new reference to the filesystem it's a reference to a merkle tree. Since they still have all the unchanged files they only have to request from IPFS the changed files.

Being new to BrowserFS, I'm unsure which abstraction to use to implement this.

Also, since BrowserFS works with emscripten, you could use a file as a socket in wasm by subscribing to changes to the file.

@jvilk:

The most direct way to implement a file system is to extend BaseFileSystem with a class that implements the FileSystem interface. Note that BaseFileSystem only implements a subset of that interface, so your file system should explicitly implement that interface. You also need to extend PreloadFile, like in the HTML5 FileSystem backend, to implement close and sync operations on open files. In this scenario, you will need to figure out a mapping from POSIX-like files/folders/metadata to IPFS.

Alternatively, you can implement the AsyncKeyValueStore interface, and BrowserFS handles mapping files/directories/etc onto the key value store. You'll also need to extend the AsyncKeyValueFileSystem interface, like in the IndexedDB file system, to specify your file system's options and to add code that instantiates your AsyncKeyValueStore.

It seems like it might be easier to directly implement the file system interface on top of IPFS (the former paragraph) if IPFS already has a concept of a directory structure.

Hope that helps! Also, please forgive my sluggish responses; I'm much less responsive than usual as I defend my PhD and move across the country in a month.

@jvilk:

Also, once you add a factory class that can create instances of your IPFS backend to the testing code, you can run BrowserFS's full suite of unit tests against your backend.

@neuronsupport:

I think new version of IPFS/JS(0.30) now supports the unix like Mutable File System (MFS) which would be helpful.

jvilk/BrowserFS#203