pkg/store: init #86
pkg/store: init #86
Conversation
It is explained in the description though ;-) It's picking up one of the ideas from https://alternativebit.fr/posts/nixos/future-of-nix-substitution/, which hashes each file inside the .nar individually. It might make sense to not involve a complicated chunker if the file is not much bigger than a defined file size, so this might entirely get absorbed into the import logic. If not, we could probably give it a better name at least :-)
It wasn't really useful - adressed in ecf6906. Each chunk now is just an alias to |
flokli
force-pushed
the
store
branch
2 times, most recently
from
cbf1958
to
e3d65a0
Compare
This provides two different implementations to chunk data.
very WIP, and not sure about some of the interfaces yet.
Chunkers should immediately return io.EOF if an empty []byte is passed to them. We don't want empty chunks to be legal.
This provides a way to hash bytes, and get back a go-multihash identifier for it.
This provides a convenience writer, which can be used to verify hashes and filesizes match some expectations.
Co-Authored-By: Jonas Chevalier <[email protected]>
…k}Entry This allows using these later with a common interface.
This converts a PathInfo back to a .nar file.
| // This means, we render the NAR file twice - once to calculate NarHash, NarSize, | ||
| // a second time to do the actual upload. |
There was a problem hiding this comment.
Maybe that's an argument for actually storing NarHash and NarSize in the PathInfo.
@adisbladis, what do you think?
This provides a bunch of store-related components and concepts. Very WIP. PR opened for some early feedback.
A "Path Store", storing information about an output path (some .narinfo-style metadata, and the listing of .nar files)
A "Chunk Store", storing content-adressed chunks
Something converting from .narinfo and .nar to that format, plus tests.
I'm not yet sure about the package layout.
pkg/nixpathfeels a bit alien, and I'd probably like to have two packages, one for "path metadata and stores", and one for chunks (?)There still needs to be a store exposing a "legacy" binary cache with these new interfaces (populating a "Path Store" and "Chunk Store" on access).
There still needs to be something converting back from these two stores to .nar and .narinfo
Once that's done, we probably want to write an HTTP handler, so
nix copyto it, andnix-store -rfrom it becomes possible.There needs to be more tests (for all chunk stores, for all path stores).