nova-cache

nova-cache implements the Nix binary cache protocol in Haskell: nix-base32,
NAR serialization (whole-tree and streaming), narinfo parsing, store paths,
Ed25519 signing and upload validation, a WAI application for a cache server,
and an optional server executable.
Bounded xz, zstd and bzip2 codecs are the public nova-cache:xz,
nova-cache:zstandard and nova-cache:bzip2 sublibraries, and the zstd codec
also compresses for uploads.
Parsing, formatting, hashing, signing, validation, one-shot xz decompression
and zstd compression are pure. Reading and writing trees, hashing files, the
streaming codecs, zstd and bzip2 decompression, the store and the server run
in IO. nova-cache is the protocol layer under
nova-nix.
Installation
build-depends: nova-cache >= 0.13 && < 0.14
The codecs are separate sublibraries, named alongside the main library:
build-depends: nova-cache:{nova-cache, xz, zstandard} >= 0.13 && < 0.14
libbz2 and libzstd are bundled. liblzma comes from the xz package, which
prefers a system liblzma found through pkg-config and falls back to its bundled
sources. constraints: xz -system-xz in cabal.project forces the bundled
copy, as this repository does.
Usage
import NovaCache.Hash (hashBytes, formatNixHash)
import qualified Data.ByteString as BS
-- Hash file contents into sha256:<nix-base32>
hash <- formatNixHash . hashBytes <$> BS.readFile path
import NovaCache.NarInfo (parseNarInfo)
import NovaCache.Signing (parseSecretKey, sign)
-- Parse a narinfo and sign it
case (parseNarInfo raw, parseSecretKey "mykey:base64...") of
(Right ni, Right sk) -> print (sign sk ni) -- Right "mykey:<base64 sig>"
(Left err, _) -> putStrLn err
(_, Left err) -> putStrLn err
import NovaCache.Validate (validateFull)
-- Validate an upload: fields, NAR hash, file hash and signatures. Every
-- error is collected rather than stopping at the first.
case validateFull publicKey ni narBytes fileBytes of
Right () -> accept
Left errs -> reject errs
import qualified Data.ByteString as BS
import NovaCache.NAR (defaultCaseHack, withNarSource)
import qualified NovaCache.Hash as Hash
-- Stream a tree's NAR and hash it in one pass, without holding the archive
-- in memory. NovaCache.NAR.Stream parses NARs incrementally, and the codec
-- sublibraries bound decompression by a narinfo's declared NarSize.
narHash <- withNarSource defaultCaseHack path $ \pull ->
let go ctx = do
chunk <- pull
if BS.null chunk
then pure (Hash.hashFinalize ctx)
else go (Hash.hashUpdate ctx chunk)
in go Hash.hashInit
Server
CACHE_API_KEY=change-me cabal run --flag server nova-cache-server -- --port 5000 --store ./nix-cache
Without CACHE_API_KEY the server refuses to start. --allow-open-writes runs
it with unauthenticated writes and removals, for local testing only.
GET / answers a line of plain text. --root-page FILE serves FILE there
as HTML instead, such as a page about the cache; the server reads it once at
startup and refuses to start if it cannot.
The protocol lives in the NovaCache.Server library module as a WAI
Application, so the cache can be embedded in another server with its own
root page. The bundled executable is one such embedding. One server serves a
store directory: it orders its own removals against its own uploads.
Configuration
| Variable |
Description |
PORT |
Listen port (default: 5000; also --port) |
HOST |
Bind host (default: all interfaces; also --host) |
NIX_CACHE_DIR |
Store directory (default: ./nix-cache; also --store) |
CACHE_API_KEY |
Bearer token for the requests marked authenticated below. The server refuses to start without it unless --allow-open-writes is passed. |
SIGNING_KEY_FILE |
Ed25519 secret key file for server-side narinfo signing |
LOG_REQUESTS |
Set to 0 to disable request logging |
Endpoints
| Method |
Path |
Description |
GET |
/ |
A line of plain text, or the --root-page file |
GET |
/nix-cache-info |
Cache metadata |
GET |
/narinfo-hashes |
All cached narinfo hashes, newline-delimited (authenticated) |
GET |
/<hash>.narinfo |
Fetch a narinfo |
GET |
/nar/<file> |
Fetch a NAR (streamed from disk) |
PUT |
/<hash>.narinfo |
Upload a narinfo (authenticated, validated; refused unless the NAR its URL names is stored with its FileSize and FileHash) |
PUT |
/nar/<file> |
Upload a NAR (authenticated, streamed to disk) |
DELETE |
/<hash>.narinfo |
Remove a narinfo; its NAR stays until a reclaim (authenticated) |
GET |
/reclaim |
List what POST /reclaim would remove (authenticated) |
POST |
/reclaim |
Remove the NARs no narinfo names and abandoned uploads (authenticated) |
POST |
/nar/<file>/uploads |
Begin a multipart upload of a NAR (authenticated) |
PUT |
/nar/<file>/uploads/<id>/<n> |
Upload part n of it (authenticated, streamed to disk) |
POST |
/nar/<file>/uploads/<id> |
Complete it: join the parts into nar/<file> (authenticated) |
HEAD is answered wherever GET is.
Uploading a NAR in parts
A proxy in front of the server can refuse a request body over its own limit,
as Cloudflare does over 100 MB on its Free and Pro plans. A NAR over such a
limit is uploaded in parts, one request each:
-
POST /nar/<file>/uploads answers 201 with the upload's id, followed by
a newline, as the body, and Location: uploads/<id>.
-
PUT /nar/<file>/uploads/<id>/<n> sends part n as the body. Parts are
numbered from 1 and go in order: part n is accepted once parts 1 to
n - 1 are held. A part sent again replaces the one held, so a part whose
response was lost can be sent again.
-
POST /nar/<file>/uploads/<id> completes the upload. Its body declares
how many parts were sent and how many bytes they hold together:
Parts: 3
FileSize: 107200936
The server joins the parts into nar/<file> only if it holds that many
parts and they join to that size. Nothing of an upload is under nar/
before then. The narinfo follows as for a NAR sent in one request.
Each part is as large as the client chooses. The parts of an upload together
are held to the 4 GB of a single PUT /nar/<file>, and an upload to 10,000
parts.
| Status |
Meaning |
400 |
The file name, the part number or the completion body is not valid |
401 |
The write key is missing or wrong |
404 |
No such upload of that file: never begun, completed, or reclaimed |
409 |
The part would leave a gap, or the server holds other parts or another size than the completion declares |
413 |
The parts would hold more than 4 GB, the completion declares a FileSize over 4 GB, or its body is over 1 KB |
A part whose response is lost can be sent again. To a client that sends its
parts in order, a 404 or 409 for a part means the server no longer holds
the parts sent before it, such as an upload a reclaim removed, so it begins a new
upload. Completion joins the whole NAR before it answers, so its
response can be lost, or a retry can meet the first attempt still joining:
whenever the completion is answered with anything but 200, or not at all, the
client uploads the narinfo, whose check of the NAR's FileSize and FileHash
decides whether it is stored, and begins a new upload if the narinfo is refused
for want of its NAR.
Removing entries and reclaiming NARs
DELETE /<hash>.narinfo removes an entry. Its NAR stays under nar/, still
served, until a reclaim removes it. A narinfo uploaded for a path the cache
holds replaces the entry, as for a smaller encoding of the same NAR, and the
NAR the old narinfo named likewise stays until a reclaim.
A reclaim removes the NARs no narinfo names, and the multipart uploads nothing
has been written to, once each is old enough. GET /reclaim lists what
POST /reclaim removes. Both answer with a report in plain text.
auth="Authorization: Bearer $CACHE_API_KEY"
curl -fsS -X DELETE -H "$auth" https://cache.example/<hash>.narinfo # remove an entry
curl -fsS -H "$auth" https://cache.example/reclaim # list
curl -fsS -X POST -H "$auth" https://cache.example/reclaim # remove
- A NAR or an upload is eligible once it is at least
min-age-hours old
(default 336, two weeks), measured from its modification time, as in
POST /reclaim?min-age-hours=24. A NAR is unreferenced until its push sends
the narinfo, and a push can run for hours. Any other query is refused with
400.
- If any narinfo cannot be read or parsed, or its
URL is not nar/<file>,
the reclaim answers 409 naming those narinfos and removes nothing: the NAR
such a narinfo names is unknown, so no NAR can be called unreferenced.
- If the store holds NARs and no narinfo at all, the reclaim answers
409 and
removes nothing: a store that lost its narinfos looks the same as one whose
every NAR is unreferenced. Removing every entry leaves a store like that;
stop the server to remove its NARs.
- A reclaim can run while pushes are in progress. A narinfo is never stored
naming a NAR the reclaim removes, and a NAR uploaded during a reclaim is
kept. A narinfo that arrives for a NAR the reclaim removed is refused, as
when a client skipped uploading a NAR the cache held, which upstream's
nix copy does; the client's next attempt uploads the NAR.
- Uploads wait while a reclaim or a
DELETE runs, and a reclaim reads every
narinfo, so on a store with many entries a reclaim holds pushes up for as
long as that read takes.
- A Nix client holding a narinfo that names a removed NAR treats it as a
missing substitute and tries its next substituter.
- A removal that fails is reported for that file, and the reclaim answers
500.
Public cache
A public instance runs at cache.novavero.ai. It serves the sources
nova-nix's Windows packages are built from and the closure of its Windows
hello recipe, which nova-nix's CI publishes on every push to its main
branch. To use it from Nix, add it to /etc/nix/nix.conf. On a multi-user
install, Nix ignores substituters set by users it does not trust.
extra-substituters = https://cache.novavero.ai
extra-trusted-public-keys = cache.novavero.ai-1:9gQ7tLWMM+2tdC9H5sKMJltDIPfD7X2GWlZe8Aa8hHQ=
Building from source
Tested with GHC 9.14.1. CI uses the latest cabal-install release.
cabal update
cabal build all -f server --enable-tests --ghc-options="-Werror"
cabal test -f server --enable-tests --ghc-options="-Werror"
License
Apache-2.0. See LICENSE and NOTICE.