feat(gist): fsspec file system for GitHub gists (resolves #888) #1791

lmmx · 2025-02-16T19:40:32Z

This PR introduces a new filesystem backend, GistFileSystem, which allows read-only access to files within a single GitHub Gist (as suggested in #888). I'd find this really useful in combination with Universal Pathlib (also an fsspec project)!

Gists are essentially flat collections of files, so there is no subdirectory concept. (Technically they are git repos that can store dirs too but we only need to support them as flat file lists, that's all the website UI will show them as)
The implementation is closely based on GithubFileSystem but simplified for a single gist.
Supports both public and private gists, latter needed user/token (PAT).

Users can do:

import fsspec

# For a public gist
fs = fsspec.filesystem("gist", gist_id="729837f14264089288178a5f632221ab")
print(fs.ls(""))  # lists files
with fs.open("test1.txt", "rb") as f:
    print(f.read().decode())

For a private gist, the same but also passing username and token args.

Implemented FS (methods: ls, _open, cat, invalidate_cache), read-only impl
Added to registry (alphabetically)
Added a test - I changed the gist ID to one by martindurant so you wouldn't have to worry about relying on someone else preserving their artifact for your tests to pass, also it's fairly small so shouldn't be slow to load
Added documentation in docs/source/api.rst.
Verified that read-only operations (ls, cat, open) are working with public gists.

Example usage

Below is a short snippet showing how to retrieve files from a public gist:

import fsspec

gist_id = "16bee4256595d3b6814be139ab1bd54e"
print("Gist ID:", gist_id)
fs = fsspec.filesystem("gist", gist_id=gist_id)
file_list = fs.ls("")
print("Files in the Gist (via fsspec):", file_list)
contents = fs.cat(["gistfile1.txt"])
print(contents["gistfile1.txt"].decode()[:120] + "\n...")

⇣

Gist ID: 16bee4256595d3b6814be139ab1bd54e
Files in the Gist (via fsspec): ['gistfile1.txt']
import astropy.io.fits._tiled_compression as tiled
from astropy.io import fits
import numcodecs
import fsspec
import zar
...

martindurant · 2025-02-21T23:34:42Z

Thanks for providing! I haven't had a chance to look yet, but I will soon :)

lmmx · 2025-02-22T13:07:45Z

Most welcome, no worries! 😃

martindurant · 2025-02-22T21:08:02Z

Quick suggestion: it would be good to enable bundling the gist ID with the URL:

with fsspec.open("gist://16bee4256595d3b6814be139ab1bd54e@/test1.txt", "rb")

like github: allows. It would require enabling extracting kwargs from the URL.

martindurant

As I read through, I found that you answered almost all of my questions elsewhere in the code :)

I don't think there's a good way to test this thoroughly, but at least we can reasonably expect gist to be available whenever GHA is running.

martindurant · 2025-02-24T14:35:22Z

fsspec/implementations/tests/test_gist.py

+import fsspec
+
+
+@pytest.mark.parametrize("gist_id", ["16bee4256595d3b6814be139ab1bd54e"])


Perhaps a good idea to pin to a specific hash, so that the values can be tested.

martindurant · 2025-02-24T14:36:12Z