rcp

A data mover for local paths, NFS exports, S3 buckets and HDFS, distributed under the Apache-2.0 license. The default build uses audited open-source cryptography crates (RustCrypto and the official BLAKE3) and otherwise only its own code; the release binary includes TLS support and uses the host’s GNU libc. There is no bundled interpreter or telemetry.

Start with building and runtime requirements and release installation and verification. The Rust API reference is published beside this book at https://rust-cp.github.io/rcp/.

rcp copy     [copy options] [-newid ID | -noId] SOURCE DESTINATION
rcp resume   [copy options] -id ID
rcp sync     [copy options] -id ID [-snap PATH]
rcp sync dry-run -id ID [-target] [-l | -q]               # what sync would do
rcp isync    [copy options] SOURCE DESTINATION            # incremental, no catalog
rcp isync estimate SOURCE DESTINATION | -id ID
rcp verify   [verify options] SOURCE DESTINATION
rcp scan     [scan options] [-newid ID] SOURCE            # or: rcp SOURCE
rcp scan     -id ID [scan options]                        # reports from a saved scan
rcp estimate [-sample BYTES] [-gbit N] [-target DIR -t DURATION] SOURCE
rcp estimate -id SCANID [-gbit N]
rcp delete   [-parallel N] [-force] [-removetopdir] PATH
rcp chown    [-v] (-user U [-group G] | -reference PATH | UID[:GID]) [-user-from U] PATH
rcp chmod    [-v] (-mode MODE | -reference PATH | MODE) PATH
rcp show     [-all] | HOST... [-v]                        # mounts, or a server's exports
rcp indexlist | indexdelete (-id ID | -match GLOB) | logdump [ID]
rcp diag     (-cat | -mkdir | -rmdir | -rmrf) PATH | -rmid ID     # file utilities
rcp diag     nio | find | change | indexdiag | rebuild | dump     # see below
rcp help [filters | info] | version

Endpoints are directories, written as /path or file:///path; NFS exports written as host:/export, host:/export:/subdir, nfs://host/export or, through several of the server’s addresses at once, ip1,ip2:/export; buckets written as s3://bucket/prefix; or HDFS paths written as hdfs://namenode:port/path. A mounted NFS or SMB export is a directory like any other; an export named directly is reached with the built-in NFSv3 client, which needs no mount and no root and keeps many requests in flight per connection; a bucket with the built-in S3 client; an HDFS path over WebHDFS; a WebDAV collection. Any endpoint can be the source or the destination of any command, with the exceptions a transport’s own model forces: a registry stores blobs and manifests rather than a filesystem, so oci:// takes no symlink, no hard link and no chmod or chown destination, and it cannot be a -nodes destination because only the coordinating process publishes; a tar:// stream is written once through and cannot be modified in place. Each of those refuses by name rather than silently doing nothing.

Install, upgrade and uninstall

Quick start

1. Install rcp (either way below).

2. Copy a tree, then check it:

   rcp copy /data /backup
   rcp verify /data /backup
   

3. Run it again later: rcp sync moves only what changed, using the catalog the first copy recorded. rcp help lists every command and option, and the command-line reference has the same tables.

From a release archive

Each release on the releases page carries an x86-64 GNU/Linux archive, SHA256SUMS, an SPDX SBOM, a Sigstore signature and build provenance (Releases shows how to verify them). Install into a prefix, using the usual PREFIX and DESTDIR conventions:

sha256sum --check --ignore-missing SHA256SUMS
tar -xzf rcp-vX.Y.Z-x86_64-unknown-linux-gnu.tar.gz
PREFIX=${PREFIX:-$HOME/.local}
install -D -m 0755 rcp-vX.Y.Z-x86_64-unknown-linux-gnu/rcp "${DESTDIR}${PREFIX}/bin/rcp"

Uninstall by removing that file: rm "${DESTDIR}${PREFIX}/bin/rcp". rcp installs nothing else. Its catalog (~/.local/state/rcp/catalog unless -catalog, RCP_CATALOG or XDG_STATE_HOME says otherwise; see The catalog) is yours to keep or delete.

Container image

The signed runtime image is public at ghcr.io/rust-cp/rcp, tagged with the version. It runs as UID/GID 10001 with /state as its home and catalog, and has no shell or package manager (container/README.md describes it). Pull it, verify it, and run it with the data mounted. Verifying needs cosign v3 or later. The release workflow signs with cosign v3, which stores signatures and attestations in the newer Sigstore bundle format, attached to the image as OCI referrers. cosign v2.x does not find them there and reports “no signatures found” for every published image, 0.1.1 and 0.1.2 included.

docker pull ghcr.io/rust-cp/rcp:X.Y.Z
cosign verify ghcr.io/rust-cp/rcp:X.Y.Z \
  --certificate-identity-regexp '^https://github.com/rust-cp/rcp/\.github/workflows/release\.yml@' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com
docker run --rm --read-only --cap-drop ALL --security-opt no-new-privileges \
  --tmpfs /tmp:rw,noexec,nosuid,nodev,size=16m \
  --mount type=volume,source=rcp-state,target=/state \
  --mount type=bind,source=/data,target=/data,readonly \
  --mount type=bind,source=/backup,target=/backup \
  ghcr.io/rust-cp/rcp:X.Y.Z copy -noown /data /backup

podman takes the same arguments. The bind mounts must be readable (and /backup writable) by UID 10001, or add --user "$(id -u):$(id -g)". Uninstall with docker rmi ghcr.io/rust-cp/rcp:X.Y.Z and docker volume rm rcp-state.

The bare binary is also published as an OCI artifact at ghcr.io/rust-cp/rcp-binary, for hosts that pull through a registry: oras pull ghcr.io/rust-cp/rcp-binary:X.Y.Z fetches rcp and SHA256SUMS (Releases).

With Cargo

From source, with Cargo’s own install convention (--root, or CARGO_INSTALL_ROOT, picks the location; the default is ~/.cargo):

cargo install --locked --git https://github.com/rust-cp/rcp --tag vX.Y.Z            # default build
cargo install --locked --git https://github.com/rust-cp/rcp --tag vX.Y.Z --features rustls
cargo uninstall rcp

TLS needs --features rustls. A source build without it has no TLS: every TLS endpoint (https://, davs://, rcps://, and S3, OCI or WebHDFS over https) is refused with an error naming the feature, and only their plaintext forms work. The release archive, the container image and the registry binary are all built with rustls; a source build that talks to anything over a network you do not control should be too. rcp version lists what a build supports.

The release profile strips symbols. For a build that keeps debugging information, Cargo’s profile variables apply: CARGO_PROFILE_RELEASE_STRIP=none CARGO_PROFILE_RELEASE_DEBUG=true. RUSTFLAGS and Cargo’s other standard variables are honored; the build adds no flags of its own beyond Cargo.toml’s release profile.

Upgrade

Install the new version over the old one, by either method above. Only the latest release is supported (SECURITY.md). Read the release’s section of CHANGELOG.md first: changes a user must act on are listed there, and the project follows Semantic Versioning, so before 1.0 a minor version can carry a breaking change and says so in its notes.

Development environment

Everything needed to change rcp and run its tests, on Debian or Ubuntu:

git clone https://github.com/rust-cp/rcp && cd rcp
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --profile minimal -c clippy,rustfmt
sudo apt-get install -y --no-install-recommends zstd krb5-kdc krb5-admin-server krb5-user libgssapi-krb5-2
python3 -m pip install --user --require-hashes -r tests/support/requirements.txt
cargo test && cargo test --features rustls

These are the same steps CI runs (.github/workflows/ci.yml). CONTRIBUTING.md lists the checks to run before pushing.

Commands

Options

-parallel N (default: the CPU count, between 2 and 8) sets the number of workers for every command that has any. -match GLOB and -exclude GLOB filter by path relative to the root: * within a component, ** across components, ?, [a-c], [!x]; a pattern without / also matches the last component, so -exclude '*.tmp' means what people expect. Directories pass an include pattern during a copy so their children can match; for chmod, chown and delete the pattern applies to the entry itself. -minsize SIZE, -maxsize SIZE, -newer AGE and -older AGE (64k, 2g; 30m, 7d, 2w) select files by size and modification age, -type f|d|l, -uid and -gid (names or numbers) by kind and owner, -fileid N by inode number; all apply to every command that walks a tree. rcp help filters has examples.

-dryrun on copy, resume, sync and isync plans, prints the counts (files, bytes, removals, skips) and writes nothing, not even a catalog. A destination that does not exist yet is planned against as an empty one and is not created. That holds on local, WebDAV, HDFS, SFTP, SMB and NFS destinations. NFS resolves the path component by component; the walk stops at the first missing one, and every call on that root then answers “no such file or directory” rather than acting on the parent it did reach. -progress prints a line every five seconds on stderr with files, bytes and rate, and once at the end; it is on by default when stderr is a terminal and -noprogress turns it off. Stdout stays parseable either way.

Copy options: -engine auto|uring|threads, -nohash (no digest; the kernel copies the bytes with copy_file_range, which is a reflink on XFS and Btrfs and a server-side copy on an NFSv4.2 mount – this is the kernel acting on an ordinary path, not rcp’s own nfs:// client, which speaks v3 and v4.0 and offers no server-side copy at all), -nolinks (hard links become separate files), -noxattr, -chown (ownership must be preserved, otherwise the entry fails), -noown (never touch ownership), -nodes HOSTS and -copybatch PROG (cluster mode, above), -preserve-atime, -fsync (per file; the default is one syncfs per job), -nocache (drop copied data from the page cache so a large backup does not evict the working set), -bs SIZE (block size, default 1m), -depth N.

Verify options: -nodata, -noattrs (no mode, owner, mtime or attributes), -noown, -nomods (no mtime), -mtimewindow SECONDS (tolerance for servers that round), -noxattr, -nolinks, -v (a verdict line per entry), -csv (differences as kind,path,detail).

-bwlimit RATE (64m, 1g, 500k/s) caps the bytes read from sources per second across every worker and engine, one token bucket for the whole run, so a migration can share a link. -loglevel DEBUG writes one line per event on stderr: each endpoint opened and its transport, an NFS server’s ports and programs, every S3 and HDFS request with its status, every worker launched in cluster mode. INFO, the default, writes none, and any other level is refused. -dircount SIZE (default 64k) is the READDIRPLUS request size on an export (on v4, the READDIR dircount). -nfs.vers 3|4 and -nfs.sec sys|krb5|krb5i|krb5p are described under NFS above. -md5 checksums the files the way -sha256 does; the digest is SHA-256 unless -blake3 selects BLAKE3 (the official blake3 crate with its SIMD kernels, several times faster per core; not a FIPS digest, which is why SHA-256 stays the default). The -csv column and {hash} placeholder follow the digest in force. Accepted and without effect: -nonames on commands without listings, -preserve-atime on commands that only read (sources are opened with O_NOATIME), and license / activate (no activation is required). -s3.endpoint, -s3.profile, -s3.insecure and -s3.noverify are described under S3.

Ownership is the one thing a non-root copy cannot fully preserve. By default a refused chown is counted and reported once as a warning; -chown turns it into a per-entry error, -noown stops rcp from trying. Extended attributes the destination refuses (trusted.* and security.* without the capability) are reported the same way.

The catalog

One directory per copy ID, at -catalog DIR, else $RCP_CATALOG, else $XDG_STATE_HOME/rcp/catalog, else ~/.local/state/rcp/catalog. Inside, index.tsv starts with the two endpoints and then holds one tab-separated line per completed entry: path, kind, size, mtime (ns), mode, uid, gid, SHA-256, source inode (the column sync uses to recognize a renamed file; catalogs written without it still read). Lines are appended and flushed as entries complete, so a copy killed part-way leaves a truthful index. A copy ID that already exists is refused: two copies must never share a resume record. jobs.log beside the catalogs records every command.

Exit codes and output are the contract

Wrappers can rely on three things: the exit code, the final STATUS: line, and ERROR: lines. rcp never prints STATUS: FAILED while exiting 0. The integration tests in tests/contract.rs pin this, and every arm that shows a check passing has a control arm that plants the defect and requires the check to fail. Sockets, FIFOs and device nodes are skipped with a WARN: line. A copy that fails on some entries still copies the rest, reports each failure, exits 1 and keeps its catalog, so resume can finish the job.

Delta transfer

-delta sends a file the destination already has as a patch against it instead of whole. The destination cuts its copy into content-defined chunks (a Gear rolling hash with FastCDC’s normalized chunking, 8 KiB on average for a file up to 32 MiB and larger for larger files) and signs each with the run’s digest; the source cuts its own copy the same way, sends the chunks the destination lacks as bytes and the rest as “copy this range of what you have”; the destination rebuilds the file beside the old one, checks the whole-file digest the source computed as it went, and only then renames it into place. The destination is signed first and then opened and checked against the size, mtime and inode the run stat’ed, so a file replaced or rewritten in between is refused; one that changes after that fails the whole-file digest check at the end, and the old file is kept either way. The rebuilt file grows under an unpredictable name created exclusively beside the old one, an interrupted patch leaves nothing behind, and a destination that is one name of a hard-linked inode gets a new inode while the other names keep theirs. Whatever stops a patch, the file is then copied whole, so -delta never fails a copy that would have succeeded without it. Files under 64 KiB and files the destination lacks move whole.

Two things about the chunking are worth knowing because both were defects first. The cut masks sit HIGH in the fingerprint word: Gear pushes an old byte’s contribution toward the high bits, so testing the LOW bits decided each cut from a 15-byte window, and two places in a file whose last fifteen bytes agree then cut identically – every record boundary in a fixed-width-record file, every run of padding. And the chunk signatures always use a 32-byte strong digest whatever -hash says, because they are a wire detail between two rcps rather than something a user reads; sharing the -hash choice made -delta -xxh3 transfer every file whole while printing output identical to a run with no -delta at all. Both changed where chunks fall, so the rcp serve protocol version moved with them and a mixed pair fails by name.

Over ssh:// both halves run where the bytes are: an ssh destination signs itself and applies the patch on its own host, an ssh source takes the signatures and matches on its own host, so what crosses the wire is signatures one way and changed chunks the other. Between two ssh endpoints the machine running rcp is the relay, so signatures and literals each cross two links; the serves never talk to each other. On a local or NFS endpoint the same code runs in process through the port’s defaults. A sparse file is chunked as zeros on both ends, so an unchanged one sends nothing and the zero chunks travel as holes. The summary says what it saved: delta: 3 files, 1085249 literal bytes, and -loglevel DEBUG gives the per-file line.

Measured on a 32 MiB file with 1 MiB rewritten: about 1.07 MiB of literals over about 3,560 signatures, in every direction. The chunker is content-defined, so an insertion shifts only the chunks around it; a fixed-block scheme would lose everything after the edit.

Wire compression codecs

-compress keeps its LZ4 behavior and its version-5 handshake. -compress.codec names a codec, lz4, deflate or zstd, and implies -compress. -compress-codec is the same option, the spelling the first zstd change used. Naming two different codecs with the two spellings is refused. The codec applies to payloads over ssh://, rcp:// and rcps://, including delta literals and streamed reads. Choosing between them is covered below (Pick zstd when…).

zstd is in every build: this crate carries its own encoder and decoder, so no library and no feature is needed. cargo build --features system-zstd swaps in the system libzstd as the backend instead. Install the development/link library first (for example libzstd-dev on Debian); the feature adds a small binding to its stable one-shot C API, with no new Cargo packages, build script or downloads. The two backends make and read the same frames, so a build with the feature and one without it compress to each other in both directions. The test suite checks this. Cargo still needs registry metadata for the optional rustls dependency to resolve the lockfile offline.

The handshake. Raw and LZ4 use protocol 5, byte for byte as before. Any other codec uses protocol 6, naming itself with a tag in HELLO: zstd is 2, deflate 3 (plus its level). The serve must answer with the same tag. A serve that does not speak 6 refuses it by version before it reads the target or creates a directory. A v6 tag it does not answer is refused too. There is no silent downgrade, so use -compress for LZ4 with an older peer. A payload’s tag is never sent unless HELLO agreed to it.

The same prefix probe, 10% saving threshold and incompressible-block pause apply to every codec. Unprofitable blocks go raw. Compressed payloads carry the original length and a CRC-32 of the original bytes, and a claimed length beyond what the codec can produce is refused before anything is allocated. The same codec also reads and writes .tar.zst through tar://; see Archives.

Compression on the ssh stream

-compress asks an ssh:// far side for LZ4 on the data payloads (reads, writes, streamed segments, delta literals; never metadata or holes, which are already small). It is agreed in HELLO and applied one block at a time: each payload carries a flag saying whether it is raw or compressed; a block is probed on its first 64 KiB and goes raw without the rest being tried when the probe does not shrink below nine tenths; after eight such blocks in a row the next sixty-four go raw with no probe, then one is probed again, and if it does not pay the pause repeats at once. So text costs a compression pass per block and shrinks, and a tree of already-compressed data settles at one 64 KiB probe per sixty-five blocks and crosses at its own size. The codec is the LZ4 block format in this crate, so there is nothing to install on either side, and a serve from before this option answers the request with a no and the run proceeds uncompressed. -loglevel DEBUG says which of the two happened.

-compress.codec deflate swaps in the DEFLATE encoder this crate already carries for .tar.gz, and -compress.level 1..9 chooses how hard it works on both sides. On 4.5 MB of log-shaped text pulled through a counting stand-in, the serve sent 793,848 bytes at level 1 and 599,604 at level 9.

-compress.codec zstd swaps in a zstd encoder and decoder written for this crate, again with nothing to install: frames carry Huffman-coded literals and FSE-coded sequences and are read by zstd -d, and the decoder reads raw, RLE, Huffman and treeless literals, one bitstream or four, predefined / RLE / FSE-compressed / repeated sequence tables, and the three repeat-offset codes. Interop is asserted in both directions against /usr/bin/zstd: five corpora at three levels, plus a boundary sweep over the lengths where the format changes shape (0, 1, the 1 KiB single-stream/four-stream literals split, either side of the 128 KiB block edge; text, single-byte runs, incompressible noise, skewed wide alphabets, and blocks whose literals are nearly one byte) with and without a frame checksum, at levels -3 and -19 – 280 reference frames in the sweep alone. The frame’s XXH64 content checksum is both written and verified: the reference recomputes it on every frame this encoder produces, and refuses one whose content has been altered. The skewed corpora are there for a measured reason – without them the sweep never once entered the FSE-coded Huffman-weight decoder, so the arms said nothing about it. Three more branches were at zero when somebody finally counted – treeless literals, single-stream Huffman and repeat-mode sequence tables, all of which parse attacker-supplied bytes and none of which any reference frame had ever reached. A large skewed corpus (several blocks in one frame, so the reference starts reusing trees and tables) and two small ones (a literals section under 1 KiB, the only size the single-stream form can describe) fix it, and the sweep now fails if any of the four stops being exercised.

Concatenated frames used to be silent data loss. Given two frames in one buffer this decoder returned the first and stopped, with no error, where zstd -d returns both – measured at 320 bytes against 700. Every frame is decoded now, bytes that are not a frame are a refusal rather than a shrug, and skippable frames are skipped in all five arrangements the reference accepts them in (first, middle, last, two adjacent, and an empty payload).

A frame that names a dictionary is refused. Stepping over the header’s Dictionary_ID is the tempting shortcut, and it is silent tolerance: the blocks after it decode, the checksum over them matches, and the decoder returns bytes on an input zstd -d will not touch – measured, the reference answers Decoding error (36) : Dictionary mismatch and zero bytes where the shortcut returned 1,024. Same class as the multi-frame defect. A hostile corpus does not catch it on its own, because its bit flips do set the dictionary flag but never insert the ID, which shifts every later field and makes both decoders refuse for unrelated reasons; a well-formed dictionary frame has to be built, and is, at all three ID widths. What is still not detectable here: a dictionary used with its ID omitted, which the format permits and which no header distinguishes from an ordinary frame – the reference cannot tell either.

A declared window over 128 MiB is refused, and that was laxness until it was measured from the other direction. Every check here had been asking what this decoder refuses that zstd -d reads. Asking the inverse – what does the REFERENCE refuse that we read – found a frame from zstd --ultra -22 --long=31 that decoded here and got Window size larger than maximum : 2147483648 > 134217728 there. The boundary is measured on both sides: a hand-built single-segment frame declaring exactly 1<<27 decodes in both, and one declaring 200,000,000 is refused by both. What it costs is said rather than hidden: the reference’s cap is a default, so zstd -d --long=31 reads frames this now refuses. That trade is made this way round because the peer running zstd -d with no flags is the common case. It costs this codec’s own output nothing – a wire frame is at most 16 MiB. Hostile input is checked against the reference too, not just for panics: over 4,710 corrupted, truncated and random inputs, 129 are accepted and byte-identical to the reference, none are accepted only by us, and four are refused only by us – all four because RFC 8878 reserves the low two bits of Symbol_Compression_Modes and libzstd ignores them. A refusal the reference disagrees with fails that arm unless it names a rule like that one. Accepting where the reference refuses fails it too, and that zero is a measurement rather than a counter nobody has seen move: removing the dictionary refusal moves it from 0 to 3, so the zero means the corpus contains inputs that would expose laxness and none of them do. A planted case would not serve: a frame with trailing bytes that the decoder read to the frame end looks like one, but that behavior is a defect, and fixing it leaves such a counter with no subject at all. The same corpus is required to reach eight named conformance refusals (it reaches 24 distinct ones in total); a check that stops firing fails by name, which is how this codec keeps finding code that was argued about rather than run. Pushing the same 4.5 MB through the counting stand-in: plain ~4,537,400 bytes, lz4 ~1,118,400, zstd ~605,000, deflate ~602,600. (The last digits move by a few bytes between runs because the count includes the request paths, and the scratch directory’s name changes; the comparison is stable, the units digit is not.)

Getting there was encoder work rather than format work, and each step was measured on this same arm: lazy matching 960,872 -> 856,164, FSE sequence tables built from the block’s own symbol counts 856,164 -> 630,260, repeat offsets 630,260 -> 605,008. A deeper match chain was tried and reverted: 64 instead of 32 doubles the search and bought 0.16%.

Engines

The copy family has two file engines and picks one at runtime.

io_uring (default where the kernel allows it): each worker owns a ring and drives up to 16 files at once as small state machines: open both ends, read a window of chunks, write each chunk as it lands, hash chunks in offset order, apply metadata, close. One thread keeps a queue depth of dozens of operations, which is what an NFS server or an NVMe device needs to show its throughput; a thread doing one blocking read then one blocking write cannot. The ring is set up with raw syscalls and hand-mapped shared memory rather than through a crate. Kernels older than 5.6, or hosts where io_uring_setup is disabled by policy (a common hardening setting, and Docker’s default seccomp profile), make the setup fail and the thread engine takes over with no difference in the result.

threads: the portable engine. Per-worker buffers, SEEK_DATA / SEEK_HOLE extent maps so holes cost nothing, copy_file_range per extent when no digest is wanted, O_NOATIME on sources, posix_fadvise(SEQUENTIAL), positional writes, metadata through the open descriptor (fchmod, futimens, fchown, fsetxattr), and a single syncfs at the end of the job.

That final flush is a durability step. It is tempting to make a failing one a warning, on the reasoning that the files are in place, named and readable, with only their writeback to stable storage in doubt. It is a failed copy, for every destination, because that reasoning is wrong: a syncfs that fails is the filesystem saying it could not write the data, which is not a weaker claim than the files being visible. The same call on a registry is a publishing step — the manifests go up there, and until they do nothing can be pulled — so there a failure fails the run. One port method carrying both meanings would apply one policy to both; the tree answers which kind it is, and that same answer is what refuses a fleet against a destination nobody would ever call sync on.

Both engines hash with SHA-256 on the x86 SHA extensions when the CPU has them (about five times the portable rate), detected once at runtime; rcp version says which. The directory walk is parallel too: a work queue of directories shared by the workers, because a metadata-bound tree is limited by lstat latency, not bandwidth.

Cluster mode

rcp copy -nodes w0,w1,w2 SOURCE DESTINATION spreads a copy over a worker fleet: the coordinator (this process) lists and plans as usual, then hands the file batches, balanced by size, to rcp copybatch workers started over ssh on each node and collects one result line per file. Directories, hard links, symlinks, metadata and the catalog stay with the coordinator, so resume -id and sync -id work on a fleet-made copy exactly as on a local one. A node needs rcp on its PATH and the same endpoints reachable: an export or a bucket is shared by nature, a local path has to be the same mount everywhere. localhost as a node name runs a worker on this host without ssh, and RCP_SSH names the ssh program. scan -nodes lists the top level here and one top-level subtree per worker, merging the entries. resume and sync take -nodes too.

-copybatch 'PROG ARGS' runs a program of yours instead of rcp copybatch on every node (or here, without -nodes), as a line protocol: the program gets SOURCE DESTINATION appended to its arguments, NUL-separated paths relative to the source on stdin, and prints one line per path, ok<TAB>PATH<TAB>BYTES<TAB>SHA256-or-empty<TAB>OWNER-REFUSED<TAB>XATTRS-REFUSED or fail<TAB>PATH<TAB>MESSAGE. A path with no line, or a worker that dies, is a failure for every file it was given; the coordinator never guesses.

IPv6 addresses

An endpoint may name an IPv6 literal in brackets – dav://[::1]:8443/p, oci://[fe80::1]/repo, ssh://user@[::1]:2222/p, hdfs://[::1]:9870/p, nfs://[::1]/export – and one reader handles the authority for all of them, so a bracketed host cannot mean one thing to the endpoint parser and another to the transport’s.

The interesting part is the arm that was missing. Unifying the parsers and proving it by round-tripping Endpoint::parse into display() and back into each transport’s own parser is not enough: every such arm passed, and dav://[::1]:8443/p still could not open. A parser round-trip is not a connection, and nothing measured the connection. The host arrived at (host, port).to_socket_addrs() still wearing its brackets, and [::1] is not an address – it is a name, so it went to the resolver and came back failed to lookup address information.

The brackets are correct everywhere else: Host: [::1]:8443 needs them, CONNECT [::1]:443 needs them, RFC 3986 requires them in the URL. So the fix is a strip at the three points of use rather than a second host field: the resolver (unbracketed in connect_tcp, which every HTTP adapter and the proxy dial share, and in the NFS RPC client – those are the crate’s only two to_socket_addrs call sites), and rustls’s ServerName, which reads an IP or a DNS name and [::1] is neither.

Two parsers were also still splitting on the last colon with no bracket awareness, each reachable only by a route the table arm did not cover: -s3.endpoint http://[::1]:9000 became a host of that whole name on the scheme’s default port, and HTTPS_PROXY did the same. Both read the authority the way the other four do now, and the table arm covers all six.

nfs:// is deliberately different: its display form is host:/export, where brackets mean nothing, and the bare spelling ::1:/export already stored the address. The two spellings disagreed – one worked and one did not – so the URL form now stores the address too, per element, since an address list may be nfs://[::1],[::2]/export.

A zone id is the second half of this. A link-local address carries one – which interface to leave by – and RFC 6874 writes it %25eth0 in a URI because % must be escaped there. Strip only the brackets and the resolver gets fe80::1%25eth0, which is not an address either, so it goes to the name lookup exactly as [::1] used to. Measured, and it needs no IPv6 stack because it is a parse: fe80::1%25lo fails the lookup and fe80::1%lo resolves to [fe80::1%1]:80. A certificate is the opposite case – a zone is a local routing detail and no part of the peer’s identity, so nothing names one and rustls refuses the whole string – and unzoned drops it for the ServerName alone.

NO_PROXY was mangling the addresses it was told to exempt. The port strip split every entry on its last colon, which is right for example.com:8080 and destroys an address: NO_PROXY=::1 became :: and NO_PROXY=[::1] became [::, so neither matched the stored host and traffic went through the proxy the operator had just excluded. A port is stripped only when what follows the colon is one, both spellings are accepted, and the comparison is made on the bare address.

What is NOT measured here, and the arm says so in its own comment: this container has no IPv6 stack at all (bind [::1] answers EAFNOSUPPORT), so no arm in this suite makes an IPv6 connection. What the arm measures is the thing that was wrong – that the literal reaches the resolver as an address and not as a name – with a control that a genuine unresolvable name still fails the other way, so the distinction it reads is a real one.

Driving rcp from an agent

rcp job runs a command in a session of its own and reports on it afterwards, so a caller need not hold a tool call open for a terabyte. rcp job start copy -newid ID SOURCE DESTINATION returns a job id at once; status, wait, log, cancel and forget work from that id from any process, including one started later. The command after start is taken verbatim, so its options are its own rather than the job command’s.

On Linux x86-64 and AArch64, job cancel uses a process handle (pidfd) so a recycled PID cannot receive the signal. It requires Linux 5.3 or later and permission to use pidfd_open and pidfd_send_signal; if those calls are unavailable, cancellation reports an error instead of signaling an unpinned PID.

Nothing there is a daemon. The job is the ordinary command with its events going to a file, in its own session so a closing terminal or a dying agent host does not take it with it. rcp job log -n OFFSET reads forward from where the last read stopped and never hands over half a line, so polling it never sees a partial event.

A canceled job leaves its catalog and rcp resume -id ID finishes it. That depends on when catalog entries are written. The catalog is what resume reads; with entries written in one pass after every file in the batch had landed, a copy killed after 28 of 40 files left no file entries at all, and resuming it copied all 40 again. Entries are written as each file lands, so a canceled job resumes from where it stopped. A contract arm measures exactly that: it counts the entries while the job is still running, cancels, and requires resume to skip that many.

resume is the remedy for a COPY, and only for a copy. It re-plans the copy against the catalog and never plans a removal, so a failed sync or isync is finished by running that verb again, and a run that kept no catalog at all (-noid, a dry run, a bare isync) has nothing to resume. Each of the three names what will actually work, in the summary and in rcp job show alike. Advice that cannot work costs an operator a run before they learn it was wrong.

rcp mcp serves JSON-RPC 2.0 on stdio, one message per line, so an agent host can call rcp as a tool instead of assembling a command line and parsing prose: initialize, ping, tools/list and tools/call, with the tools rcp_plan, rcp_copy, rcp_isync_plan, rcp_isync, rcp_verify, rcp_scan, rcp_count, rcp_delete, rcp_show and rcp_describe. Each returns the -json events of the run it made.

Every destructive tool has a plan tool beside it that runs the dry run of its own command, and the pairing is not decoration: a copy’s plan lists every source file, an isync’s lists only what differs and what it would remove, and the digest is a hash over those lines. While rcp_isync was told to take rcp_plan’s digest, the two printed different numbers for the same pair of trees (20d6219d81bec157 against 296677cd670b40fc) and the call could not be satisfied at all. A test walks every destructive tool, requires the command it plans with to be its own, and then runs the pair end to end.

Every call runs rcp itself as a child. Running the command in the server’s own process would be shorter by a page and wrong: the command writes its events to stdout, which there is the protocol channel. A child keeps the two apart, gives every call the same argument parsing and the same rails a person gets, and means a copy that dies takes nothing with it.

Two things follow from the child, and both were measured rather than assumed. A killed server takes its child with it: without that, killing the server left the copy running, so a host that had given up on the call was still being written to — the child asks the kernel for a death signal when its parent goes, and a contract arm kills a server mid-copy and requires the child to be gone. (rcp job is how a run is meant to outlive its caller, and it puts itself in a session of its own so this never reaches it.) And a reply is bounded: the child’s events are held whole in memory to be returned as one string, so a scan of a hundred million files would be a hundred million events in a single reply. Past 4 MiB the middle is dropped, with a warning event saying how many went and both ends — the beginning and the done — kept, since a tool call is the wrong shape for a firehose and rcp job is the right one.

The rails are the point of the wiring rather than an afterthought. The server’s rails reach every call – -allow, -readonly on the command line, and its -oci.user, -s3.endpoint, -s3.profile, -hdfs.namenode and transport switches through the child’s environment – and -requireconfirm is added whether or not it was asked for, so a call that would remove or overwrite is refused until its confirm carries the digest rcp_plan (or rcp_count) returned for the same endpoints and options. A copy into empty space just runs, because whether a call is destructive is something only the plan knows and the child computes it. A call’s options are checked against this build’s own option tables, so a call cannot invent an option, smuggle an endpoint through them, set the rails itself, or turn certificate verification off for a host the operator named.

Say what does NOT reach a call, because “whatever the server was started with” is not the rule. -nodes, -copybatch and -rcp.remote name a program to run or a host to run one on, and are relayed neither way: no rail bounds what a program does, so a server started with one does not hand it to a caller and a caller cannot ask for one. -catalog IS relayed, as RCP_CATALOG, so the children share the directory the server was started with (see relay_env). -nfs.sec, -nfs.vers and -acl4 are carried, through RCP_NFS_SEC, RCP_NFS_VERS and RCP_NFS_ACL4; the first two are Transport and a call may not name them at all, and -acl4 a call may add but cannot strip. A hand-written summary of a table the code already publishes is easy to get wrong, and this one has been, more than once: run rcp describe rather than trusting this sentence.

Checking the option NAME is not enough. -copybatch takes a program to run per batch — a documented feature, and the right one for a person at a terminal. Through the server it was arbitrary command execution: a rcp_copy call carrying -copybatch 'touch /tmp/rcp-pwn-probe', to a server started with two -allow roots, created that file.

A list of the nine options whose value is a capability would be the same shape as the bug: a tenth would be exposed until somebody remembered a second table. So every option now carries a classification in the same table that carries its summary — Setting, Runs, Path, Service, Credential, Transport or Rail — with no default variant, and tests/describe.rs fails the build when an option the parser accepts has none. A call may set what is classified Setting and nothing else; the rails read the same table for the options no rail can bound (-copybatch, -nodes, -rcp.remote), which are refused on any railed run whether or not a server is involved. rcp describe publishes settable_by_a_call per option, so an agent reads the rule instead of learning it from a refusal. No count is written here, deliberately: a count that once stood in this place said “100 of the 115 options are settable, 15 are not” while the table had already grown to 117 options and 23 refusals. A number copied out of a table is a second copy of the table. Ask the build:

rcp describe | grep -c 'settable_by_a_call.*false'

Three rails bound a run, all off by default, because a person at a terminal should not have to ask twice. A caller that drives rcp on someone’s behalf sets them and then cannot exceed them, whatever it was told to do.

-allow ROOT (repeatable, or RCP_ALLOWED_ROOTS, comma-separated) is the set of roots the run may touch. Every endpoint must be a root or lie under one, on a path boundary, so /data does not admit /database; a local root never admits a remote endpoint, or the reverse. The check happens before anything is opened, so a refused endpoint costs no connection and learns nothing about the far side.

A local endpoint is resolved the way the kernel resolves it before it is compared. A spelling check is not containment: with only a spelling check and -allow /data in force, /data/../secret passed the rail and its files were copied, and so did /data/link where link was a symlink to anywhere at all. Both were measured, both are refused now, and the arms that cover them fail if the lexical check comes back. Components are resolved from the root outwards, each existing one canonicalized before the next is applied, because that is the order the kernel uses — a/b/.. is not a when b is a symlink. What does not exist yet is appended, so a destination that has never been created is still checked. Symlinks inside a tree need none of this: a copy recreates them and never follows one.

A resolver like that fails open easily. canonicalize fails for a path that does not exist and for a symlink whose target does not, and taking the spelling in both cases admits ROOT/out with out a dangling link to anywhere — a create follows a link. A component that is a link is read and followed whether or not its target exists (40 hops, then it stops, as the kernel does); only a path with no link left in it is taken as spelled. Two more of the same kind: -allow / is a root that admits every local path, where trimming its trailing slash naively leaves an empty string that is dropped, leaving a run with no roots — a rail that is not there. And a remote endpoint whose text climbs (nfs://host/export/../secret) is refused rather than compared: only the far side can resolve it, so there is nothing honest to compare. sftp:// and s3:// already refused one when they parsed it; this is the rest.

A rail on the text of an endpoint is not a rail on the service it resolves to. rcp mcp refuses every option that is not a plain setting; a railed command line refused only the two that name a program to run. So this went through:

$ rcp scan -allow s3://corp/bucket -s3.endpoint http://127.0.0.1:8931 s3://corp/bucket

and a listener on loopback — a host no rail named — received GET /corp/?list-type=2&prefix=bucket/, while the same run without -s3.endpoint goes to AWS. -hdfs.namenode is the same shape, and -s3.profile and -oci.user choose whose credentials ride along. A railed run now refuses all four on the command line.

What that does is move the capability to the environment, not remove it, and it is worth saying exactly. RCP_S3_ENDPOINT redirects the same traffic, and so do AWS’s own AWS_ENDPOINT_URL_S3, AWS_ENDPOINT_URL and an endpoint_url in the named profile, which rcp reads because AWS’s own tools do. The rails bound the command line — the part a caller writes and an operator reads — and not the environment a run inherits. Where those are the same party this changes nothing. Where they are not, which is the case the rails exist for, the caller writes the command line and the server owns the environment.

-snap and -reference stay, because they are opened through the same -allow check as any endpoint; the rails themselves stay, because on a command line they are the operator’s own.

The registry’s upload hand-off is bounded by the same roots. A registry answers the upload POST with a Location, and for ECR, GAR and anything fronting object storage that names a different host, which is where each blob’s bytes then go. Following it is the feature. But the registry picks that host, and -allow is the operator saying where this run’s data may go, so a railed run follows the hand-off only to an authority its own roots name, and says which -allow would permit it. An unrailed run follows it as before: naming the registry is already the statement of trust. Never a loop — the hand-off is read at most twice, once when the upload opens and once when it commits.

What is still true and worth saying plainly: this is a check before the open, so it cannot close a race against a symlink planted between the two, and it constrains endpoints rather than every syscall a run makes. It is a rail, not a sandbox.

A symlink inside the destination is the other half of that, and it is not a rail question at all — it was wrong for a copy with no rails on. A destination is not always a tree this run made, and a link sitting where a file belongs was followed: measured, a copy into a tree holding f.txt -> /elsewhere/target wrote the source’s bytes into /elsewhere/target and left the link in place, and a linked directory took a whole subtree with it. Both halves are wrong — the write escapes and the copy does not produce what the source has. Local destinations open with O_NOFOLLOW now (both engines, since each opens its own), and a link found in the way is removed and replaced by the file or directory the source has. A symlink in the source is still recreated as a symlink, and an ordinary overwrite still overwrites; both are controls on the same test. The separator is a comma rather than the colon a PATH uses, because every endpoint scheme has a colon in it: splitting on colons turned ssh://host/path into the two roots ssh and //host/path, which admitted nothing at all. A test control caught that.

-readonly (or RCP_READONLY) refuses every command that writes to an endpoint, before it starts. Which commands write is the table rcp describe publishes, so the two cannot disagree – plus three whose write-ness depends on an ARGUMENT rather than their name, which the table cannot express: logdump ID writes ID.zip, scan -newid ID writes a catalog, and estimate -target DIR copies a live sample into DIR to time it. All three reported PASSED under -readonly before they were listed; finding one such command is the cue to look for its siblings.

The boundary this draws is the ENDPOINTS, stated exactly because the looser version was false: a read-only run still appends to jobs.log under the catalog directory, and a run railed with -allow /only-here still writes <id>/index.tsv under the catalog too. That is rcp’s own record of what ran, in a directory the operator named with -catalog, and suppressing it would trade an audit trail for a slogan.

That is easy to read as a hole in -allow rather than a boundary, so the reason it is safe is worth stating rather than leaving to be re-derived: catalog is itself a Rail, and a call through rcp mcp cannot set it. A railed COMMAND LINE can – -allow /data -catalog /elsewhere is accepted, because why_not_railed admits Rail. That is not a hole, and the distinction is the whole point: the rails bound what a caller may choose through a command line the operator wrote. Whoever writes -catalog on the command line is the operator, and the operator may put the catalog anywhere. The property that holds is that nothing reaching rcp through a call can move it. -allow bounds where a caller may send the DATA; it is not a sandbox around the process, and a rail that could be pointed anywhere by the thing being railed would not be one.

-confirm DIGEST with -requireconfirm (or RCP_REQUIRE_CONFIRM) refuses a run that would remove or overwrite anything unless the digest matches the one a dry run of the same command printed on its plan: line (for delete, the counting run without -force). The digest covers both endpoints and every planned action, so a change that alters the plan since it was read no longer matches. It names the actions and is not a snapshot of either tree: new content under the same planned overwrite, or a change outside the run’s scope, leaves the plan and its digest as they were. The refusal deliberately does not name the digest it wanted: a refusal that handed it over would let a caller satisfy the check without ever reading the plan, which is the only thing the check is for. A test asserts the refusal does not contain it.

One consequence worth stating: a plain copy never asks what is already at the destination, which is what makes it fast. When the plan is going to be shown or confirmed it has to ask, because not looking is the same as not knowing, and a rail that cannot see an overwrite does not guard against one. So a dry run of a copy onto an existing tree now marks those files * rather than +, which is what it always should have said.

Two more things the rail sees. The destination is listed through the path patterns (-match, -exclude) only, so a destination file that -minsize, an age or an owner filter would leave out still shows as the overwrite it is. The attribute filters are then applied to decide what the run may skip or remove, so such a file is never removed. And a destination directory the listing could not read is reported as a failure and shown on a ? line, and it makes the run count as destructive: a rail cannot rule out an overwrite it could not see.

A directory the source dropped is removed as an rmdir on every transport, including WebDAV, whose collection DELETE is recursive by the RFC and is only sent once the collection lists empty. One residual on WebDAV only: a name written into that collection in the single round trip between its listing and the DELETE goes with it, where an rmdir would refuse. The protocol has no non-recursive delete to send instead. In a filtered or -depth-capped run, such a directory can still hold a name the run did not look at. It stays, with that name inside it, and a WARN line says so; that is not a failure.

The attribute filters (-minsize, -maxsize, the ages, -uid, -gid, -type, -fileid) are tested against each side’s own entries. The source is listed through the path patterns too: a file the source still has, outside the predicates, is neither copied nor removed, the way rsync’s --min-size skips a transfer rather than hiding a name. Where a predicate describes the destination copy differently from its source, the two never meet. That is -uid or -gid without ownership carried, an age where the store keeps its own times, and -fileid always. Such files are recopied on every run and never removed.

isync and sync replace an entry of the wrong kind: a file or symlink where the source has a directory is removed before the directory is made, and a directory where the source has a file joins the removals, children first. Both show on the plan as # lines. copy and resume replace the first kind too, since one entry replaced is the same size of act as a file overwritten. A copy that did not list its destination finds the entry when the mkdir fails, looks at it without following a link, and replaces it then. A directory where the source has a file stays a failure for them: removing it would remove a subtree the source never had, which is a mirror’s decision. The symlink case matters beyond convergence: left in place, the copy would open through it and write wherever it points. Any entry in the way counts, including one the filters would otherwise leave out.

That holds wherever the client can see the link. A local O_NOFOLLOW open sees it, and so does SFTP’s LSTAT; where a server leaves the permissions out of LSTAT, READLINK tells a link from a directory. NFS never follows one at the name: an UNCHECKED CREATE hands back the link itself, and a v4 OPEN answers NFS4ERR_SYMLINK, so the reply is read and the link replaced. SFTP v3 has no open that refuses to follow, so a link planted by a concurrent writer between the look and the open is still followed. More generally, each replacement looks and then acts, so an entry another writer puts at the name in between can be the one removed. A destination that something else is changing during the run is outside what these checks settle. NFS replaces a link whether the server hands it back from CREATE (Linux) or refuses the CREATE with EXIST. INVAL is any bad argument, so a link found beside it is left alone. v4 replaces one only on the NFS4ERR_SYMLINK status, never after another refusal. Where SFTP’s LSTAT carries no permissions, READLINK is the only evidence taken: a name it answers for is a link and is replaced, and any other name is unknown. OPENDIR follows a link, and draft-02’s REALPATH need only normalize the spelling, so neither can tell a directory from a link to one. On such a server an existing directory fails its mkdir, and the subtree beneath it is refused loudly. Nothing is made, moved, removed or linked beneath a directory whose mkdir failed, since what stands at that path is not the directory the plan made. An SMB2 server without the POSIX extensions resolves links itself: Samba with follow symlinks presents a link as the file it points at, with that file’s size and times. To the client that name is a regular file, as indistinguishable from the target as a hard link is, and an overwrite goes where the server sends it, as it does for any SMB client. Where the link may point is the server’s setting. Samba’s wide links, off by default, keeps it inside the share, which is not the same as inside the destination directory. A Windows server reports a link as a reparse point instead. The listing names it as not copied, and the server refuses to open through it.

rcp describe prints what this build can do as one JSON object: every command with the arguments it takes and whether it writes, every option with its kind and a one-line summary, every endpoint scheme with a capability table (writable, sparse files, hard links, symlinks, extended attributes, ACLs, rename, identity, delta transfer, and the finest modification time it can carry), the -json events, the error codes and the exit codes. So a caller does not have to be told out of band what this binary accepts, and does not have to parse the usage text to find out.

It cannot fall behind the binary, and that is the point rather than a hope: the option list is built from the very arrays the argument parser matches against, and tests/describe.rs fails the build when an option has no summary, when a summary names an option the parser no longer accepts, when a described command is not in the dispatch, or when describe and version disagree about the transports. Each of those arms has a control.

The capability table is a statement about each protocol, not about a particular server: a false means the protocol has no way to say it, so rcp refuses or approximates it in the open. It is also the answer to “will this copy survive the trip”: a tree with hard links bound for sftp://, s3:// or oci:// arrives as separate files, and describe says so before the copy rather than in its report.

-json turns the run into events: one JSON object per line on stdout and nothing else, with every human line, STATUS included, moved to stderr. So a caller parses stdout whole and still gets a readable log by watching stderr. The exit code does not change: 0 for STATUS: PASSED, 1 for STATUS: FAILED.

deferred is not a failure; kept is. It is tempting to give both one answer – “kept is not a failure, and a run that kept everything still says passed” – on the reasoning that a registry delete which removed the manifests and left the blobs to garbage collection has done the thing a registry delete means. That reasoning is right about the registry and wrong about a locked WebDAV file, which is a different thing: the registry WILL remove its blobs and there is nothing left to do, while the locked file is still there and will stay there. Ask what a caller reading only the exit code concludes, and for the second case one shared answer gives it “the wrong thing”. So the two are split. A delete or sync that ends with anything kept exits 1 and says which entries the store refused; one that ends with entries deferred exits 0 and says which the store will finish. diag -rmdir and -rmrf answer the same way.

And kept covers every store. It is easy to answer kept only for WebDAV and HDFS, because those two report a refusal inside a successful response. A local EACCES, an NFS3ERR_ACCES, an SFTP permission denial and an S3 403 are the same event – the store answered and said no. Reporting any of them as a failure of the run gets the exit code right and the REMEDY printed beside it wrong, since a failure says to use resume and resume never plans removals, so the advice could not work however often it was followed. Removed::or_refused maps a PermissionDenied to kept; anything else – a timeout, a broken connection, an answer that could not be read – is still a failure, because it is one.

Every adapter calls it, not only the ones with a protocol-specific refusal. The easy mistake is to handle only that refusal: WebDAV’s 423 and partial 207 while an ordinary 401/403 drops into failures, WebHDFS’s namenode false but not a 403 over HTTP, the registry’s 405 but not a refusal. WebDAV, the adapter the whole idea came from, is as prone to it as any. All eight call it, and the WebDAV arm has a forbid_delete stand-in that measured red without it.

Keying on PermissionDenied alone is too broad. It cannot tell a 401 from a 403, because all four kind_of_status tables map both to that one kind. So a bad password would become kept, whose printed remedy is “run sync again once the entry can go” – advice that cannot come right, since the run is not authenticated at all, every remaining removal answers the same way, and a rerun with the same credential reproduces it exactly. The same defect one status further in.

The four HTTP adapters decide from the status, in net::http::delete_refused, and not from a kind that has already thrown the difference away: a 403 or a 423 is kept, and a 401 (or a 407, the same sentence about the proxy) is a failure that names the credential as what to fix. The stand-in has an unauthorized_delete switch, and the arm was measured red against the kind-based version, printing KEPT: .: the store refused: .: HTTP 401.

Local, NFS, NFSv4 and SFTP still go through or_refused, and the reason is easy to get wrong in both directions.

It is tempting to say those protocols “have no 401”. That is false: RPC rejects a call with AUTH_BADCRED, AUTH_REJECTEDCRED, AUTH_TOOWEAK and two RPCSEC_GSS problems, and rpc.rs maps all of them to PermissionDenied. From that it is equally tempting to conclude NFS has the same defect as the kind-based HTTP adapters.

That conclusion is also wrong, and the arm written to prove it is what shows so. An RPC rejection never reaches or_refused: self.call(..)? in the v3 client and self.compound(c)? in the v4 one return it first, so the only errors that arrive at the match are NFS statuses — which is exactly what or_refused is for. A delete against an export that answers AUTH_TOOWEAK already fails, already names -nfs.sec, and reports nothing as kept. A guard added on the strength of the second reading would be dead code.

The arm is an_nfs_call_refused_by_rpc_is_a_failure_and_not_a_kept_entry, with a stand-in that rejects only REMOVE and RMDIR (rejecting every call means the mount never lands, so the removal is never reached) and an honest server as its control. Without such an arm, two readings of the code can disagree indefinitely.

The arm for this uses a stand-in S3 service with a refuse_deletes switch rather than a chmod, and that is not a convenience: the suite runs as root, where unlink inside a mode-500 directory SUCCEEDS, so a permission arm built that way would pass without ever reaching the code it claims to measure.

An error’s code is one of notfound, permission, exists, unsupported, invalid, timeout, interrupted, disconnected, unreachable, nospace, quota, busy, wouldblock, io, owner (ownership could not be preserved and -chown made that an error) or usage (the arguments, an option, or an endpoint’s shape). The set is closed: an error kind with no name of its own reports io rather than inventing one, so a caller can branch on the code and be sure a future release will not hand it a name it has never seen.

busy exists because the cost of the fallback is concrete, and the measurement is the whole argument: without it, a registry answering 429 because it is over its rate limit and a repository that does not exist both arrive as io, and those call for opposite behavior — one is worth waiting on and the other never will be. 429, 502, 503 and 504 from a registry are busy, and a contract arm reads the code off a throttled stand-in. quota went in beside it and is not measured: it is what the kernel’s EDQUOT already maps to, so a local write over a quota reports it rather than io, but nothing here has produced one. Said plainly because the two are not the same kind of claim.

A closed set is a promise not to invent names silently, not a promise never to name anything: rcp describe publishes the list, so a caller reads it rather than hard-coding it, and an addition shows up there.

The entry events come from the same records the catalog keeps, so what an agent sees and what resume and sync will later read are one list, not two. scan emits an entry per item only when a listing was asked for (-l, -csv, -html, -fmt); a bare scan emits its summary alone, since a scan of a large tree is the listing and an agent that wanted it would have said so.

NFS without a mount

rcp copy /data 192.0.2.10:/backup speaks NFSv3 to the server itself: the MOUNT protocol for the root handle (over TCP, found through the portmapper), then LOOKUP, READDIRPLUS, READ, WRITE, CREATE, MKDIR, SYMLINK, LINK, SETATTR, REMOVE, RMDIR and COMMIT with AUTH_SYS credentials (the calling uid and gid, so the export’s root-squash rules apply as they would to a mount). Every worker thread holds its own connection from a pool; a reader keeps eight READs ahead of the consumer and a writer keeps sixteen UNSTABLE WRITEs in flight and commits once at the end, which is what a kernel mount’s per-mount request cap does not allow. Holes are never written, so a sparse image stays sparse on the server. Sources are copied to exports, exports to local paths, and exports to exports, all through the same commands.

POSIX ACLs travel on the wire through the NFSACL sideband program (100227, the one Linux nfsd and ONTAP serve beside NFSv3): GETACL on the way out, SETACL before the mode lands on the way in, so verify compares them and sync repairs them the way it does locally. A server without the program gets no ACL calls and the attributes are counted as refused in the summary, never dropped in silence. RCP_NFSACL_PORT names the port when the portmapper cannot be asked.

-acl4 adds NFSv4 ACLs: with it, every file and directory on an export lists system.nfs4_acl, read with an NFSv4 GETATTR of the ACL attribute and written with a SETATTR of it under the anonymous stateid, both stateless operations that need no client id, open or lease, so one small COMPOUND rides beside the v3 data path. The attribute’s XDR is what the kernel exposes as system.nfs4_acl on a v4 mount, so an export and a mount of it compare equal byte for byte. It is meant between exports, or between an export and a kernel v4 mount; a plain local directory lists no such attribute, so verify -acl4 between it and an export reports the ACLs the export has and the directory does not, which is the truth. A server that does not serve NFSv4 is refused before anything is planned rather than dropping what the option asked for. So is -acl4 -noxattr (and -acl4 -noattrs): these ACLs ride in the extended-attribute channel, so asking for them while turning that channel off carries none, and the combination used to be accepted in silence – a run that printed the same counts and the same STATUS: PASSED as one that had carried every ACL.

Kerberos: -nfs.sec krb5, krb5i or krb5p (or RCP_NFS_SEC) reaches every export with RPCSEC_GSS instead of AUTH_SYS, which is what an export with sec=krb5* requires. rcp carries no Kerberos code of its own: it loads the host’s GSSAPI library (libgssapi_krb5, the one the kernel’s rpc.gssd, mount.nfs and curl use) at run time, so the host’s krb5.conf, ticket cache (kinit first, or KRB5CCNAME) and keytabs apply unchanged and the build links no Kerberos library; a host without the library gets a clear refusal when Kerberos is asked for. Each connection establishes its own context with nfs@host, every call’s header carries a MIC, and krb5i checksums the body while krb5p encrypts it, in both directions; the pipelining is untouched, since a context is per connection and the sequence numbers are per call. The mount, the NFSACL program and the -acl4 sidecar ride the same contexts. The contract suite runs all three flavors against a stand-in that requires Kerberos, through a real MIT KDC it starts itself in a throwaway realm, with AUTH_SYS refused as too weak and a missing ticket refused before anything is planned.

-nfs.vers 4 (or RCP_NFS_VERS=4) moves the data over NFSv4.0 (RFC 7530) instead of v3: no MOUNT protocol (the export is walked from the server’s pseudo root), one client id per run established with SETCLIENTID and confirmed, a fresh open-owner and an OPEN, OPEN_CONFIRM and CLOSE per file, READs and WRITEs under that open’s stateid (still pipelined per connection, UNSTABLE writes committed once), a delegation returned the moment a server presses one, and the lease renewed when the run has been idle. Owners travel as strings: numeric while the server takes them, name@domain after its first NFS4ERR_BADOWNER (RCP_NFS4_DOMAIN names the domain, else the one the server itself uses; a name this host does not know becomes 65534). -acl4 rides the same connection as one more attribute. Minor version 0 only: no sessions (4.1) and no server-side copy (4.2), and no user extended attributes on the wire in any version rcp speaks (they are reported as refused). The contract suite runs copy, verify, sync, scan, chmod, delete, sparse files, hard links, -acl4 and Kerberos over v4 against a stand-in whose client ids and open stateids are real, so a READ under a stateid no OPEN produced is refused there as it would be on a server.

Limits, stated plainly: v3 and v4.0 only, and the v3 path is the default. show HOST asks a server for its RPC services and exports before anything is planned. RCP_NFS_PORT and RCP_MOUNT_PORT bypass the portmapper for a server on fixed ports, RCP_PORTMAP_PORT moves the portmapper. A server with several interfaces is written ip1,ip2:/export: the mount goes through the first address that answers and the connection pool rotates across all of them, skipping one that refuses, which spreads the load across the server’s interfaces.

The contract suite exercises the client against an in-process stand-in server (src/nfs/testserver.rs, a directory served over the real wire format), so the protocol code is tested on every run; behavior against a production filer is a separate check to make on one before trusting it with a migration.

Remote hosts through ssh, and without it

ssh://[user@]host[:port]/path is a directory on a host that has rcp installed. The endpoint opens ssh host rcp serve and everything a local path can do happens there: the listing, the reads and writes, hard links, symlinks, attributes, all through the same Tree port the other adapters implement. The connection is OpenSSH’s, so its keys, agent, ProxyJump, ControlMaster and host-key policy apply unchanged; RCP_SSH names another command in ssh’s place (the contract suite uses a script that runs rcp serve in place), and -rcp.remote PATH names the rcp on the far side when it is not on PATH. One ssh session per worker: -parallel 8 opens eight, which is cheap under ControlMaster.

-compress compresses the data payloads on the stream, and -compress.codec chooses how: lz4 (the default, fast), deflate or zstd – the two slower ones land within half a percent of each other and both send about 1.85x fewer bytes than lz4 on log-shaped text. The codec is negotiated in HELLO, and the serve must answer with the one asked for. A serve that cannot is refused rather than followed: two ends disagreeing about how to read a payload is a corruption, and a quiet fallback would hand the caller a stream it did not ask for (see Wire compression codecs above for the versions). Naming a codec implies -compress.

Pick zstd when the other end already speaks zstd, which is what it buys: the frames are ordinary zstd frames that zstd -d reads, and the decoder reads anything the reference writes. It is a complete pure-Rust implementation – Huffman-coded literals, FSE-coded sequences built from each block’s own statistics, repeat offsets, lazy matching. On ratio it is a coin flip with deflate on this corpus, so there is no reason to switch for that alone.

-compress.level N sets how hard deflate works, 1..9 as zlib means it, default 6. Unlike the codec it needs no agreement – a DEFLATE stream carries its own code tables, so the decoder never has to know what the encoder was told – so it rides in HELLO purely as a request for effort and reaches the far side’s encoder as well as this one. Level 6 is byte-for-byte the encoder that shipped before the knob existed, which is asserted rather than intended. Neither lz4 nor this zstd encoder has a level, so naming one without -compress.codec deflate is REFUSED rather than quietly ignored – a run that exits 0 having done nothing with the number is the failure, not the refusal.

Inside the session rcp speaks its own framed protocol, one request per sequence number, answered in order. A copy streams a file’s content as data and hole segments, so a sparse file crosses the wire as its data plus lengths and lands sparse in either direction, and a writer keeps sixteen writes in flight; a listing crosses in batches, so a tree of any size fits the frame limit; and the Read path that verify takes keeps eight megabyte reads in flight ahead of its consumer, so a round trip is paid once per window rather than once per block (a seek discards the window). The protocol mirrors the port rather than a shell: no quoting, no tar, no cpio, and an error on the far side comes back as the same io::Error kind it was there.

The trust model is ssh’s, and it is worth saying what that does and does not give. The serve runs as the user ssh logged in as, on any absolute directory the client names, and refuses a relative path that would step outside it (.., or an absolute path, is an error before anything is touched). That is the same standing an interactive login has, so it is no escalation; it is also no confinement, and a forced-command deployment that wants one needs a wrapper that fixes the root, the way rrsync does for rsync. The -rcp.remote string is passed to the remote login shell as rsync’s --rsync-path is, which is what lets it say -rcp.remote "sudo rcp", and which means a path with spaces or shell characters needs the quoting that shell expects. Numeric uid and gid cross as numbers and mean whatever they mean on the far host, which is rsync’s default too, and the hard-link and catalog identity of a remote file is its device and inode there, consistent within one tree and meaningless across two.

rcp serve is the far end. It takes no arguments, reads frames on stdin and answers on stdout, and serves the directory the client names, creating it when the client is a destination. It is what the delta and compression work rides on: a channel rcp controls at both ends.

Without ssh: rcp:// and rcps://

rcp serve -listen ADDR holds a socket and speaks the same frames over it; rcp://host:port/path is the endpoint. No child process and no handshake per channel, which is what -nodes and -via want across a tunnel something else already authenticates. An rcp endpoint takes no user (there is nothing to check one against) and no default port (there is no registered one, and inventing one means a typo dials some other service).

What the socket does not inherit is the point. Over ssh://, OpenSSH has authenticated the peer and started the far side as that user before a frame moves; a socket inherits nothing. So -listen refuses to start without -allow ROOT (or RCP_ALLOWED_ROOTS) naming what it may serve, and refuses any address that is not loopback.

rcps://host:port/path is that socket inside TLS, in a build with --features rustls. The serve presents -tls.cert with -tls.key; the client checks it against -tls.ca (the system bundle by default) and presents its own certificate with the same two options. Only -tls.clientca lets a serve bind anything but loopback, and that is deliberate: a server certificate encrypts the stream and proves who the serve is, while establishing nothing about who connected, so a serve with a certificate and no client verifier on a routable address is still an unauthenticated remote filesystem. Naming the authority the peer’s certificate must come from is the thing that changes.

serve  $ rcp serve -listen 0.0.0.0:7712 -allow /srv/data \
           -tls.cert serve.crt -tls.key serve.key -tls.clientca ca.crt
client $ rcp copy /local rcps://filer:7712/srv/data \
           -tls.ca ca.crt -tls.cert client.crt -tls.key client.key

The four options are Transport capabilities: a railed command line may set them, an MCP call may not, and rcp mcp relays the server’s choice to every child through RCP_TLS_CA, RCP_TLS_CERT, RCP_TLS_KEY and RCP_TLS_CLIENTCA.

Relay: -via ssh://hop

rcp copy -via ssh://bastion oci://registry.internal/app /local opens the registry on the bastion and copies from there. -via names an rcp serve that opens every remote endpoint of the run on this side’s behalf – a registry, a bucket, a WebDAV server, a second rcp serve – and this side speaks to it exactly as it would to a directory on that host. A chain is just an ssh:// or rcp:// endpoint behind a -via: the hop opens it through its own transport. -via takes either kind of serve. Local endpoints stay local, which also means a relay is only exercised when the endpoint behind it is remote.

Two things about the trust model are the whole design. The hop opens the endpoint with its own environment: its credentials, its transport settings, its RCP_OCI_INSECURE, because a relay’s secrets are the hop operator’s and a HELLO is not the place to carry someone else’s. And the hop operator has a rail: RCP_ALLOWED_ROOTS in the serve’s environment (an authorized_keys environment=, ~/.ssh/rc, a wrapper) or -allow on a -listen serve bounds what any client may ask that serve to open, endpoints included. That rail bounds the endpoint TEXT, so a root of /srv admits /srv/x and does NOT admit rcp://filer:7712/srv/x – a hop that is to open remote endpoints needs them spelled in its own roots. Unset, a serve opens what the ssh user can reach, which is what it always could. -via itself is a Runs capability: a call through rcp mcp cannot set it and a railed command line strips it.

The hop’s HELLO reply also carries what the far tree answers the planner – whether it keeps times, whether its metadata can be rewritten in place – because the SshTree on this side is what the planner asks, and without them it would answer the defaults. Without that override, sync -via ssh://hop to a WebDAV collection would recopy every file for one edit, the same bug keeps_times fixes on the direct path, and chmod through the hop would be accepted and silently not done. Both are test arms, with a local directory through the same hop as the control.

Every worker’s channel is its own rcp serve, and each one opens the endpoint again. Only the first open asserts that the endpoint is there. The channels after it send ATTACH instead of HELLO: the same body and reply, and no “is there an image here” check, because the run’s own work can make that answer false for a tree that is fine. A delete removes the last tag before a slow channel comes up, and a push has not published its first tag yet. Both used to fail on the late channels, as “the repository has no tags” and “no such repository”. A join keeps the first open’s push rights and never creates anything, so a second channel cannot recreate a directory the run removed or start a second archive. A serve that predates ATTACH answers it Unsupported, and the pool falls back to the old HELLO on the same channel once, then sends HELLO to that pool for the rest of the run.

A registry behind a hop is written through one channel. A registry publishes its manifests at sync, and each serve stages what it received, so files written through a second channel sat in a staging area the first channel’s sync never reached. Measured through a relay at -parallel 8, the push reported success and the tag did not exist. The engine now writes such a tree one file at a time, the way it writes a tar archive, for the same reason -nodes refuses a registry as its destination. Reads are not limited.

Hosts with only OpenSSH: sftp://

sftp://[user@]host[:port]/path is a directory on a host where rcp is not installed but OpenSSH is: the endpoint opens ssh host -s sftp and speaks SFTP version 3 to the subsystem, one session per worker, with reads kept eight deep and writes sixteen deep. Everything above works across it: copy, sync, verify, scan, delta (the signing and matching run on this side, since the far side is only a filesystem). What the protocol cannot say is refused or approximated in the open: no inode numbers, so a hard link on an sftp source is a second copy of the file and one bound for an sftp destination is written as a file (hardlink@openssh.com is used where the server offers it); times to the second (verify accepts a whole-second destination against a source with nanoseconds); no holes, so runs of zeros are detected on read and skipped on write; no extended attributes, so every one is reported refused. The trust model is ssh’s, as for ssh://, and the same RCP_SSH hook reaches a stand-in for the suite.

Each read and write is the size the server says it takes. A server that offers limits@openssh.com (OpenSSH 8.7 and later) is asked, and OpenSSH answers 256 KiB less 1024. One that does not gets 32 KiB, which every version 3 server must accept and which OpenSSH’s own client uses in the same case. -loglevel DEBUG prints the sizes in use.

This used to be 256 KiB for every request, on the belief that it was the largest read or write OpenSSH accepts. OpenSSH’s 256 KiB limit is on the whole message, header included, so every file of 256 KiB or more failed against a real server. Measured against OpenSSH 10.3: 262,100 bytes crossed and 262,144 did not. The suite’s stand-in is paramiko, which accepts messages of up to 16 MB, so it could not see the problem. It now enforces OpenSSH’s limit and offers the extension as OpenSSH does, and it can also leave the extension out, as a server older than 8.7 does.

A session that ends mid-file is a failure the run reports. ERROR names the file, the run ends STATUS: FAILED, and the catalog keeps what completed for rcp resume. Before, a server that dropped the connection killed the run on its next write: exit 141, no message.

Windows, Samba and NAS shares: smb://

smb://[user@]host[:port]/share[/path] is an SMB2 share. The first path component is the share — what TREE_CONNECT takes — and the rest is a path inside it. Dialects 0x0202, 0x0210 and 0x0300 are offered; 0x0311 deliberately is not, because it requires the preauth integrity hash this client does not compute — signing, which it does, is a separate matter and is covered below. Authentication is NTLMv2, with the credential in -smb.user [DOMAIN\]USER:PASSWORD or RCP_SMB_USER with RCP_SMB_PASSWORD — never in the URL, which reaches the job log, a report and ps, and a URL carrying one is refused without being quoted back.

Copy, sync, verify, scan, delete and rename all work across it, in both directions. What a share cannot say is refused or reported rather than invented: there is no POSIX mode and no owner, so those are synthesized for the listing and verify reports them as not compared instead of holding an invented 0644 against a real 0755; no symlinks and no hard links, both refused by name (the core copies the file instead); no extended attributes; and no hole map, so runs of zeros are detected on read and skipped on write, as on sftp. Modification times do cross, both ways, through FileBasicInformation — at SMB’s own resolution, which is the 100-nanosecond FILETIME tick, so a local nanosecond time arrives rounded down to it. verify treats that as equal, the way it already treats a destination that keeps only seconds or milliseconds. Precisely: the guard is “either side sits on a tick”, not “both do” — it has to be, since the case it exists for is a nanosecond source against a tick-aligned destination. An SMB destination is always on a tick, so against a share this does accept any source within the same 100 ns bucket. A file written one tick later is still a difference; one written 40 ns later is not. That is the same shape as the whole-second tolerance beside it and a thousand times narrower.

One consequence is worth stating because it is a real cost rather than a limitation of the protocol: chmod and chown refuse against an smb:// tree — correctly, since neither could change anything — and diag change -mtime gates on the same predicate, so it is refused too even though SMB2 could carry it. A copy still sets times.

Message signing is implemented, which is what decides where this works. It is tempting to read the dialect choice as though signing were a cost of 3.1.1. It is not: signing has been in SMB2 since 0x0202, and the server decides whether to require it — Windows 11 24H2 and Windows Server 2025 require it by default, as does a Samba configured server signing = mandatory, which is usual for a domain member. Against such a server there is no dialect to fall back to, so until this existed the client read the bit and refused by name, which is correct and excludes most current deployments. The client now reads the same bit and signs: HMAC-SHA256 truncated to 16 bytes on 0x0202 and 0x0210, AES-128-CMAC on 0x0300 with its signing key derived through the SP800-108 counter-mode KDF. When the server does not require signing, nothing is signed.

On a signed session every final reply is checked, and one carrying no signature is refused exactly like one carrying a wrong signature: a reply that simply clears the flag is the same tampering. Two exceptions. An unsigned error is reported as the status it claims and marked as unauthenticated; it fails either way, and real servers send them when they have no key to sign with. It carries no meaning, though: its error kind is a neutral one, so a forged OBJECT_NAME_NOT_FOUND cannot make a file read as absent, which a sync’s removal step would act on. The same holds for both SESSION_SETUP replies, which no key can authenticate on any session: their failures are reported as a refused login or as a neutral error, never as a missing file, so a forged status during login cannot make a destination read as empty. On an unsigned session none of this applies: every reply is the server’s word only, so a sync whose removals should not rest on unauthenticated answers wants -smb.sign. And an interim STATUS_PENDING is not verified, since the spec leaves it unsigned; nothing is taken from it but a wait and a capped credit grant. The final SESSION_SETUP response is checked by hand against the key the login derived, which catches a server that accepts a password it cannot check and an edit of the dialect the server chose. It does not prove which server answered: without channel binding, NTLM can be relayed to another server that accepts the same credential, and that session verifies. Nor does it catch a downgrade made by striking dialects from this client’s request, since both ends then agree on the weaker one.

On its own, that is not a defense against a downgrade, though it is easy to read it as one. Signing is armed by a bit in the NEGOTIATE response, which arrives before any session exists and cannot be authenticated. An attacker who clears that bit gets a client that signs nothing and checks nothing. -smb.sign (or RCP_SMB_SIGN=1) closes that: the client sets SIGNING_REQUIRED in its own SESSION_SETUP, which makes a server that merely enables signing sign the session anyway, arms the key whatever the server’s bit says, and refuses a server that will not sign. Measured through a relay counting reply flags against a Samba with server signing = auto: without the flag, every reply but the final SESSION_SETUP came back unsigned. With it, the only unsigned replies were the four first-round SESSION_SETUP answers, which come before any key exists, and one interim STATUS_PENDING, which the spec leaves unsigned. Because an interim reply cannot be authenticated, the credits one request’s replies can add are capped at what that request asked for. That bounds each injection; repeated on every request it can still walk the balance past the server’s real window, and what that buys is a request the server refuses, which an on-path party could get more simply by dropping the connection. It is off by default because it puts a MAC over every byte where nothing asked for one. FSCTL_VALIDATE_NEGOTIATE_INFO, which would also protect the negotiate’s other fields, is not implemented.

Signing uses the RustCrypto aes and cmac crates, and hmac with sha2 for the 2.x dialects. aes runs on AES-NI (or the ARMv8 AES instructions) where the CPU has them and on a constant-time software cipher otherwise. Measured on 1 MiB with AES-NI: about 1.7 GB/s. That is the MAC alone. RCP_NO_AES_NI, which used to force rcp’s own portable cipher, is still accepted and has no effect. A signed write copies its payload four times on the way to the socket, and a call-shaped benchmark of that path ran at about 1019 MB/s, which is below the 1250 MB/s a 10 GbE link carries. The signatures are pinned to vectors computed by an independent implementation (impacket’s KDF, Cryptodome’s CMAC), because every other arm compares this code with itself. Replanted, the KDF’s context one byte short at the call site left all of those arms green.

Measured against a Samba 4.19.5 with server signing = mandatory, at each of the three dialects this client offers — so both algorithms ran, not just the one the default negotiates: scan of 2114 files, a 213 MB copy out, a copy back in, and a verify of the two, 0 differences and 0 signature complaints in the server’s log. Still not measured: any Windows server.

Two things about getting there are worth recording, because both failed in the same uninformative way — STATUS_ACCESS_DENIED or a dropped connection, which is exactly what a wrong password looks like:

• The KDF input was one byte short. MS-SMB2 3.1.4.2 passes SP800-108 the NUL-terminated strings "SMB2AESCMAC\0" and "SmbSign\0", and SP800-108 then adds its own 0x00 separator after the label — so there are two zero bytes between label and context, a terminator and a separator that happen to be the same byte. Writing it as “label, NUL, context, NUL” gives a completely different key. The unit arm in place at the time asserted that changing the label changed the key, and it passed the whole time: a self-consistency check cannot see a missing byte. The arm that replaced it spells the input out and has the original defect replanted as its control.
• The final SESSION_SETUP request must go out unsigned. For an initial, non-binding authentication the session key exists on the server only once that request’s NTLMSSP AUTHENTICATE has been processed, so signing it produced NT_STATUS_INVALID_HANDLE and a server that could not sign its own error reply. NTLMSSP_NEGOTIATE_SIGN and ALWAYS_SIGN, meanwhile, have to be in the NTLMSSP NEGOTIATE as well as the AUTHENTICATE; with them only in the second, Samba set up no session key at all.

Both were found by reading the server’s log — it prints the signature it expected beside the one it received — rather than by reasoning about the client. Neither was visible from this side of the wire.

A real Samba is what found the one defect the stand-in could not see, and it is worth saying what each measures. The suite’s stand-in is impacket, which negotiates 0x0202 — a dialect with no credit window, where the header’s CreditCharge is reserved and ignored. This client wrote 0 there for every dialect, so the whole suite was green while any file over 64 KiB failed to reach a real server: Samba answers STATUS_INVALID_PARAMETER, which surfaces as “the server could not parse the request” and names nothing about size. Measured against Samba 4.19.5 on dialect 0x0300: 64 KiB crossed, 64 KiB + 1 did not, with MaxWriteSize reported as 8 MiB — so it was never a cap, it was the charge. It is fixed, with a unit arm on the formula at the boundary, and a copy, read-back and byte-for-byte comparison of a 3 MB tree against that Samba now passes. The lesson is the one this repo keeps paying for: the probe and the real path took different routes, so the probe could not see the failure.

And the charge is computed from the PAYLOAD, which took reading the specification to get right — a capture could not settle it. An audit of 545 captured request/response pairs showed a 1 MiB read’s response weighing 1048656 bytes on the wire, so charging the data length alone looked like a one-credit under-charge whenever the length is a multiple of 64 KiB. It was “corrected” to header + fixed part + data, and this document recorded the question as unsettled: 17 might be generous, Windows sends 16, and over-charging cannot fail where under-charging can.

MS-SMB2 3.1.5.2 settles it against that fix. The charge is computed from “the payload size (the size of the data within the variable-length field of the request) or the maximum size of the response”. A header is not payload. 3.3.5.2.5 has the server compute its expected charge from the same definition and fail only when its number exceeds what was sent — which is exactly why 17 was accepted everywhere and why nothing ever caught it, at a cost of one credit and one message id per read.

The audit measured the right bytes and answered the wrong question. “How big is this response” is not “what is this request’s payload size”, and no amount of wire capture distinguishes them. scripts/smb-audit-credit-charge.py had the same mistake baked in and would have gone on reporting a spec-correct client as under-charging, so it now computes from the payload and prints the message-sized number beside it. Re-measured against that Samba after the fix, over 401 pairs: READ and WRITE send 16, need 16, and the message-sized number is 17 — 0 under-charged, with the transfers accepted.

A connection occasionally dies during its handshake when several are opened at once, and the client now notices and dials again. Against Samba 4.19.5 about one connection in forty-five never gets an answer to its SESSION_SETUP; the server reports client used more credits than granted, mid 1, charge 1, credits_granted 0, seqnum low/range: 1/0 — its credit window empty on a connection whose own NEGOTIATE response granted a credit.

The wire says the client is not at fault in what it sends. Across 256 captured connections, wedged and healthy alike, the NEGOTIATE and SESSION_SETUP messages hash to one value each — byte-identical, same length, same CreditCharge. Every wedge has the same shape: three messages, then nothing. So there is no message to correct, and the honest fix at this layer is to notice and reconnect:

• the handshake runs on a 10 s fuse rather than the 60 s the data path needs, because a NEGOTIATE either answers promptly or is not going to. That number was measured entirely against a Samba on loopback, and a share over a satellite hop or a congested VPN can legitimately need longer — so -smb.handshake SECONDS (or RCP_SMB_HANDSHAKE) moves it, and 10 is only what it is when nobody says;
• a handshake that fails at the transport is retried on a fresh connection, up to three;
• the retry gives up on a server that is simply gone. It exists for a race that clears on a second connection — one failure in forty-five, independent. When the server is stopped or firewalled instead, every failure is correlated and the retry turns each worker’s one doomed connection into three plus two sleeps, so a pool spends its time dialing a host that is not there and the operator waits longer for the same error. After 12 consecutive failures across the pool the retry switches off: one attempt, no sleep. Twelve is well clear of the independent case. If failures were independent at 1-in-45, twelve in a row would be about a 1-in-10^19 coincidence. They are not entirely independent, though: the wedge happens when several connections open at once, so its failures cluster. Read the figure as “far from anything the race was measured to do”, not as a probability. One success re-arms it for everyone, and a refused credential never counts, so one bad password cannot switch the retry off against a server that is answering fine. The arm counts connections against a real listener rather than reading the counter, and was checked by removing the breaker and requiring it to fail;
• a refused credential is never retried. That is the half that could do harm: three attempts per connection across a worker pool is how an account gets locked out by a client trying to be helpful. The retryable kinds are named, the default is not to retry, and an arm pins it with the STATUS_LOGON_FAILURE mapping it depends on;
• a login the server hung up on is retried, but only inside the credential budget below. No error kind can tell a server refusing a password by closing the socket from one that lost a race, and Samba’s wedge was measured striking after the AUTHENTICATE as well as before it: a build that declined to retry there failed a file that the previous build had copied. The price is that a server restarting mid-login can spend the whole budget in one open;
• the password is offered at most twice in a row without being accepted, per account, for the whole run. A slot is reserved once the challenge is in hand, just before the AUTHENTICATE, and released when the server answers, so sixteen workers starting together cannot overshoot. A login that stalls before its credential holds no slot. (Taking the slot before dialing would let a stall at NEGOTIATE hold it until the handshake deadline, with every login to the account queued behind it.) A worker that waited is checked before it writes: if the server closed the connection meanwhile, nothing is sent and nothing is counted. Each offer then gets one deadline for its whole answer, however the server paces it, so a server answering a byte at a time cannot hold a slot for longer. The same deadline bounds the send: a peer that stops reading and lets this client’s socket buffer fill — draining the AUTHENTICATE a byte at a time, or not at all — cannot hold the slot open any longer than one that stalls the reply, because every write is cut off at the same clock. What an answer means is classified, from MS-ERREF’s values: only STATUS_LOGON_FAILURE is a rejected password — it counts and is not retried; an answer about the account itself (locked out, disabled, expired, outside its logon hours, a workstation or logon-type restriction) counts and stops offering that credential to the server that gave it, with the reason in the refusal — another server still gets a try, because a logon-hours or workstation restriction is that server’s own, and even a lockout seen at one domain member need not bind a different one; an answer that never judged the password (a discarded session, no reachable domain controller, a server out of resources) counts and is retried like a dropped connection. Anything else — STATUS_ACCESS_DENIED among it, which a share ACL or a firewall can raise without the password being wrong — counts but is named in the refusal as the status it was, not reported as a bad password, and is not retried. (It is tempting to call ACCESS_DENIED a rejection, or to stop the account run-wide on any account verdict; both are too broad.) The budget is reserved before an offer goes out, not checked and incremented after the fact: checking afterward let sixteen wrong passwords through against a budget of three. Two, because the Default Domain Policy ships with lockout disabled and DISA’s STIGs require a threshold of three or fewer, and a budget at the threshold is not a budget. The budget is shared by every share the run reaches under one account, keyed by the bare account name so CORP\alice and alice@corp.example are one budget. Failures are recorded per credential and offers are gated on their sum, so a right password for one alice cannot clear a wrong one’s failures: a single shared count would let every success on a copy’s destination re-arm two more wrong offers on its source. A guest session for a named user counts as the rejection it is. On a signed session only a success whose signature verifies resets the budget; an unsigned or mis-signed STATUS_SUCCESS counts against it, because that is what a forger covering a real rejection would send. That protection needs -smb.sign: without it, the same forger can clear the server’s signing bit first, and then there is no signature to check. A success clears only failures that settled before it was reserved, and never an account latch: a signature proves a success is genuine, not that it is newer than a lockout or rejection another connection reported while it was in flight. An answer already in hand when an offer was reserved is the one ordering the server’s processing is known to follow. So a transient failure before a real success does not spend the budget, while a stale success – forged, or genuine but delayed – clears nothing. A latched credential cannot be reserved again, so the latch holds for the run. (Letting a success clear it is tempting, and letting only a signed one clear it looks safer; both are the same hole.) The budget does not span runs. Past it, the refusal says which of rejections, account verdicts, hang-ups, unauthenticated answers or unrecognized statuses it saw, with advice to match – an account verdict (a lockout, a disabled account) that a different credential meets at the summed budget is named as that, not counted among “rejected”, the same distinction drawn for an unrecognized status;
• the backoff is jittered, so workers that fail together do not retry together;
• the TREE_CONNECT that follows a login gets its own full handshake deadline, not whatever milliseconds the login’s offer had left. The offer’s deadline is a budget for the AUTHENTICATE alone; letting it carry into the next request would time out a tree connect for no reason but a slow login before it, so the clock is reset before it. That reset is an absolute deadline, not the per-syscall socket timeout a plain set_timeout leaves: the latter lets a peer trickle the tree-connect reply a byte at a time and hang the worker, since the deadline-bounded reader falls back to read_exact when no absolute deadline is set. The tree connect holds no budget slot, but a small fixed exchange should not hang;
• a connection whose request went out but whose reply did not come back whole is never used again – not for the CLOSE a failed read or write used to send on it, and not by the pool. After a timeout mid-READ the rest of that reply is still arriving, and the next request would read it as its own answer. On an unsigned session those bytes can be file content, which frames as a reply as readily as anything else. A matching message id is not a framing guard, so the connection refuses further requests instead of trusting one;
• a reply must carry the response flag as well as the request’s id and command: a request reflected back matches both, and on a signed session its signature verifies, since one key signs both directions;
• a file that ends before the size its open reported is an error (“the source changed during the copy”), not a short file reported as whole; a READ returning more than it asked for, or data placed inside the response header, is refused; a listing whose pages stop naming anything new (a server ignoring the cursor) is refused, a name listed twice is listed once, and an empty name is refused; a name a path cannot carry – an unpaired surrogate, which NTFS allows, or a path separator – is reported as not copied while the rest of the directory lists; and -fsync sends an SMB2 FLUSH before the CLOSE. A destination whose stream came up short is cut to the bytes that arrived, not left at the declared length with a zero tail.

And the client now keeps a credit balance. CreditResponse sits at bytes 14..16 of every reply and nothing in the adapter parsed it — so the client asked for credits on every request, never learned what it was granted, and could send a request charging more than it held. It found out as STATUS_INVALID_PARAMETER, which says nothing about credits and is the same status half a dozen other causes produce. The balance is a diagnostic rather than a throttle — one request at a time, I/O capped well inside the window — but when it does run short the error now names the number: “this request needs 16 credits and the server has granted 1”.

A connection that dies before TREE_CONNECT has written nothing — no handle, no partial file, no operation for a second attempt to duplicate. That is what makes this safe where retrying a WRITE would not be.

Measured with an interleaved control on an identical harness, 2048 trials each:

That first measurement confounded two changes: the retry arrived together with the shorter deadline, and the harness bounded a wedged trial at 45 s — so “fail fast enough that three attempts fit inside the cap” explains the numbers just as well as “the retry works”. Settled by varying only the retry, with the deadline held at the data path’s 60 s and the cap raised to 240 s:

The retry is what does the work. The short deadline decides what it costs, not whether it helps.

The one survivor is not explained. The obvious account — the harness’s own 45 s cap landing on a trial already into its third attempt — does not add up: three 10 s attempts plus 50 and 100 ms of backoff finish near 30 s, inside the cap. Either something in that run ignored the deadline or all three connections wedged. It is one event in 2048 and was not chased; it is recorded rather than explained away.

Two things worth keeping from getting there. A harness has to reproduce the failure at a high, stable rate before any candidate can be judged — the first one ran a whole tree copy per trial, so it produced a handful of results per minute and a rate that appeared to drift between 1% and 9%; replacing the copy with a scan (the handshake and almost nothing else) gave 1024 trials a minute and a rate steady at 2.3 / 2.3 / 2.1%. Three candidates had looked promising against the noisy one and none survived the steady one: 1 credit on NEGOTIATE as Windows does, SecurityMode = SIGNING_ENABLED as MS-SMB2 requires, and a 10 ms pause after the negotiate response. And the reference-client comparison has to use the same dialect: smbclient picks 0x0311 by default and a clean run there says nothing about the 0x0300 path this client takes. Forced to -m SMB3_00 it is clean 48 of 48, so the conclusion stands — on the second measurement, not the first.

scripts/smb-decode-pcap.py is the tool for looking at what actually crossed.

SecurityMode claims SMB2_NEGOTIATE_SIGNING_ENABLED, which MS-SMB2 2.2.3 states as a MUST and which this client sent as 0 — not a legal value — until it was read against the spec rather than against servers that tolerated it. That bit is a statement of capability: the neighboring SIGNING_REQUIRED is deliberately left clear, because a client that asks for a signed session it cannot sign is worse off than one that does not ask.

The security buffer carries raw NTLMSSP rather than a SPNEGO wrapper. That excludes less than it sounds like — Windows accepts an unwrapped NTLMSSP message, and Samba’s SPNEGO layer recognizes the signature and falls back — so the servers it does exclude are stacks with no such fallback, and Kerberos-only deployments where NTLM is disabled and SPNEGO would not help either. No sealing, no Kerberos, no SMB1, no DFS.

S3 without an SDK

rcp copy /data s3://backups/site-a -s3.endpoint http://minio.lab:9000 speaks the S3 REST API with AWS Signature Version 4, over a small HTTP/1.1 client in the standard library: ListObjectsV2, HeadObject, GetObject (with ranges), PutObject, multipart uploads, CopyObject and DeleteObject. Credentials and region resolve the way the AWS tools resolve them: AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN / AWS_REGION first, then the profile named by -s3.profile or AWS_PROFILE in ~/.aws/credentials and ~/.aws/config (including its endpoint_url and region). -s3.endpoint URL (or AWS_ENDPOINT_URL) names a MinIO, ONTAP S3 or other service, addressed path-style; with no endpoint the bucket is on AWS, addressed virtual-hosted in its region. A -s3.endpoint carrying user:password@ is refused by name, as the WebDAV and http endpoints are and for the same reason, and the refusal does not repeat the credential – an endpoint reaches the job log, a report and ps, so a message that quotes it back writes the password into the log on the one path that exists to keep it out. -s3.insecure picks plain HTTP when the endpoint names no scheme, -s3.noverify skips certificate verification (and says so), AWS_CA_BUNDLE names a private CA, and HTTPS_PROXY / HTTP_PROXY / NO_PROXY are honored, CONNECT for TLS and absolute URIs for plain HTTP.

A bucket has no inodes, so the tree is spelled out the way s3fs spells it: dir/ zero-byte objects are directories (a directory that only exists as a key prefix is listed too), a symlink is an object whose body is its target, and modification time, mode, uid and gid ride in user metadata (x-amz-meta-mtime as seconds.nanoseconds, mode as the full st_mode in decimal, uid, gid). A tree copied into a bucket and back comes home with its times, modes and owners; verify between a directory and a bucket compares all of them. Hard links have no representation: the second name becomes a server-side copy (no data moves), verify reports the lost link unless -nolinks, and extended attributes are reported as refused.

Uploads stream. The executor declares each file’s size up front, so an object up to 64 MiB is one PUT with a known Content-Length and a larger one is a multipart upload whose parts open as the bytes arrive, sized so that any object fits in the service’s 10,000 parts; a sparse source’s holes are written as zeros because objects have none. Nothing is buffered beyond the socket, whatever the object size. Reads are one streaming GET per file, reissued with a Range on a seek. Listing is ListObjectsV2 in pages of 1,000 followed by one HEAD per object in parallel for the metadata, which is the honest cost of carrying POSIX metadata in a bucket; scan -q on a million objects is a million small requests.

TLS is the one thing not in the standard library. It is behind the rustls cargo feature (rustls with the ring provider), so the default build carries no TLS stack and speaks plain HTTP, which is what a MinIO on a kit or a gateway on localhost offers, and refuses an https:// endpoint with a message naming the feature. cargo build --release --features rustls needs the registry once (the lockfile pins the versions) and produces a binary that speaks to AWS and to a MinIO behind a private CA.

The contract suite runs the client against an in-process stand-in service (src/s3/testserver.rs) that verifies every request’s signature and payload hash, so a signing mistake fails a test here; the signer is also checked against the worked examples in the S3 developer guide. A real service (AWS, MinIO, ONTAP) is a separate check to make before trusting a migration to it.

HDFS over WebHDFS

rcp copy /data hdfs://namenode:9870/backup/data speaks WebHDFS, the REST face every Hadoop namenode serves, on the same HTTP client as the S3 adapter: LISTSTATUS, GETFILESTATUS, OPEN and CREATE (following the namenode’s redirect to a datanode, so the datanode names the cluster advertises must resolve from here), MKDIRS, DELETE, RENAME, SETPERMISSION, SETOWNER and SETTIMES. hdfs:///path takes the namenode from -hdfs.namenode HOST:PORT, HDFS_NAMENODE or RCP_HDFS_NAMENODE; https:// namenodes need the rustls build. Authentication is pseudo (user.name, from HADOOP_USER_NAME or the calling user) until a namenode answers 401 with WWW-Authenticate: Negotiate, the secured cluster’s challenge; then one SPNEGO token for HTTP@namenode from the ticket cache is sent (the same GSSAPI library and ticket cache as for NFS), the hadoop.auth cookie the namenode issues is kept only for later requests to the exact issuing scheme, host and port. It is not forwarded to other origins or downgraded HTTP connections. The datanode hops use the delegation token the namenode puts into its redirects, as a secured cluster expects. No ticket means a refusal that names what the namenode wanted.

What HDFS itself cannot hold: hard links (the second name is written again; -nolinks says so up front), symlinks (every current release refuses CREATESYMLINK, and rcp reports each one as a failure rather than dropping it; -type f leaves them out), extended attributes, and sub-millisecond times (verify and sync treat a millisecond-truncated time the way they treat rsync’s zeroed nanoseconds: the same when the dropped part is zero). Ownership is by name there, resolved through this host’s passwd and group files in both directions. The contract suite runs the client against an in-process stand-in namenode and datanode backed by a directory (src/hdfs/testserver.rs); scripts/interop.sh hdfs runs it against Apache Hadoop, and Hadoop’s own client reads back what it wrote.

WebDAV: dav://

rcp copy /data davs://cloud.example/remote.php/dav/files/me/backup speaks WebDAV (RFC 4918), the protocol behind Nextcloud, ownCloud, SharePoint, Apache mod_dav and most “map a network drive over HTTPS” appliances: PROPFIND with Depth: 1 walks one collection at a time (never infinity, which servers commonly refuse and which is unbounded in memory), GET with Range reads, PUT writes, MKCOL, DELETE and MOVE for rename. There is no hard link: COPY writes a second copy of the bytes, which is not a second name for one file, so hard_link is refused as unsupported and the transfer layer copies – the same bytes that COPY would have written, with the summary now saying copied rather than linked. The same is true of s3://, oci://, hdfs:// and sftp://. dav:// is plain HTTP and davs:// needs the rustls build. Credentials go in -dav.user USER:PASSWORD or RCP_DAV_USER with RCP_DAV_PASSWORD; a URL carrying user:password@ is refused by name rather than quietly stripped, because the endpoint lands in the job log, in every report and in ps. The refusal does not repeat the URL: quoting the endpoint back would put the password in the job log on the one path that exists to keep it out. The same holds for the http:// endpoint, for -s3.endpoint, and for rcp show, which takes an NFS host and would otherwise hand a mistyped URL to the resolver with the password inside the error. All four refuse without repeating the token.

Three things the protocol cannot hold, each refused by name rather than dropped: symlinks (there is no representation for one; -type f leaves them out), hard links, and any of the modes, owners and times this crate would set. That last one has a consequence for sync that is worth knowing: a server’s getlastmodified is when it wrote the file, and the property is read-only, so the destination cannot carry the source’s mtime. The planner’s “does the destination still hold this file” check used to compare size AND mtime, and read every file on a WebDAV destination as changed – measured as two files crossing for one edit before keeps_times existed. A destination that says it cannot keep times is now matched on size, which is rsync’s --size-only and the honest best such a store allows; the catalog still carries the source mtime, so a changed source is still noticed. What that cannot detect is a file edited in place on the server to the same length.

verify asks the same question, so the two commands do not contradict each other on exactly the stores keeps_times exists for. The flag’s other consumers are the relay’s HELLO, the AnyTree delegation, the planner and sync -dryrun -target; a verify that ignored it would see a correct, complete copy to a WebDAV destination held done by the planner and report an mtime difference on EVERY file. The comparison is skipped when the destination keeps its own times, and said out loud in a warning rather than dropped quietly – a check that silently stops running is how a suite goes green without measuring anything. The modes are still compared, so verify against a WebDAV destination still reports what the server cannot hold; -noattrs is the comparison this transport can make without any of that.

The XML reader is deliberately not a general one: it reads a 207 Multistatus and nothing else and refuses a DOCTYPE outright rather than defending against entity expansion – each stated in src/dav/xml.rs with the reason it is safe to omit here. It DOES resolve namespaces. It is tempting to strip prefixes and compare local names, on the reasoning that every server binds D: to DAV:, but that reasoning is about servers while the reader is fed by whatever answers on the port: an href in another namespace placed after the real one would be read as the entry’s path. Only DAV: elements are read, whatever their prefix, and a foreign element takes its subtree with it.

One consequence of the size-only match is worth its own sentence: isync – Mode::Mirror, no catalog – to a WebDAV destination skips a source file edited in place to the same length, because without a catalog there is no recorded source mtime to compare against and the destination cannot hold one. sync -id sees the edit through the catalog; isync does not. verify -noattrs after an isync is the check. S3 is the neighboring case: a bucket this crate wrote carries the source mtime in object metadata and keeps_times is honestly true, but a bucket written by another tool carries only the store’s Last-Modified, and against one of those isync recopies every file whose length matches and whose date does not. The contract suite runs the client against an in-process stand-in (src/dav/testserver.rs) that is deliberately awkward in the two ways real servers are: its listings include the collection’s own entry, in either position, and every entry carries a second propstat answering 404 for a property it lacks. An adapter that only meets a tidy server is one that has not been tested.

That was measured the hard way. Against Apache 2.4 with mod_dav, every run failed at open with HTTP 301, and the stand-in had never shown it. A collection named without its trailing slash, as an endpoint URL is written, gets a 301 from Apache to the same path with the slash. The adapter refuses redirects, and the stand-in answered 207 to both forms. Now the adapter follows exactly that redirect, when Location names the same resource on the same origin with a slash added, and refuses every other. The stand-in answers the way Apache does. Against Apache 2.4 httpd:2.4 (Debian), a tree of 10 files and 22 MB copies up and back down, including names with spaces, +, %, non-ASCII characters and a 200-character name. verify reports 0 differences, and diff -r against the server’s disk is clean.

Web servers as sources: https:// and http://

rcp copy https://mirror.example/pub/dataset /data/dataset
rcp copy -http.index '' https://mirror.example/pub /data/pub   # ignore any index; use the autoindex

A read-only tree, and the refusal is the interesting half: copy INTO one, sync, delete and chmod all fail by name rather than issuing a PUT that some servers accept and others do not. A plain web server has no agreed way to create anything; the protocol that does is dav://, which is already here.

HTTP has no directory listing. What it has is three conventions, and which one you get is decided by an operator you do not control, so all three are read, in order of how much truth they carry:

The first is this crate’s own: rcp scan -json > rcp-index.json at the tree root, published beside the files, needs no post-processing — the reader takes the newline-delimited stream the scan writes, done event and all. It is the only shape that carries the source’s own mtime, so it is the only one for which keeps_times is true; against the other two the planner matches on size and verify skips the mtime comparison and says so.

The HTML reader is where the judgement is, because an autoindex page is mostly links that are not files, and turning one into a file is how a copy acquires something called ?C=N;O=D. Four kinds are dropped and each has its own reason: a link with a query or fragment is a control (a real ? in a name arrives as %3F, so cutting on the literal loses nothing); a link that leaves the directory is navigation, which is what Apache’s “Parent Directory” is and what makes a naive walker climb out; a link equal to the directory is the self-link, the same trap WebDAV’s own PROPFIND entry sets; and a duplicate is Apache emitting the icon link and the text link for one entry.

The contract arm serves ONE fixture all three ways and compares the three results against each other as well as against the source. That is the arm that a shared omission cannot pass: a bug dropping the same file from all three would satisfy any per-shape assertion.

One limit is worth stating rather than discovering: no command drives a seek on a source reader, so the Range path here is measured by unit arms that seek the reader directly, as the registry and WebDAV readers are. diag nio is the one core caller of Seek and it lays out its own files with create_file, so it needs a writable tree and can never be a read-only endpoint’s benchmark.

Container images: oci://

oci://host[:port]/repository[:tag] is a repository on a container registry (the OCI Distribution API), presented as an OCI image layout: oci-layout, index.json and blobs/sha256/<digest> for every blob the image reaches, manifests and indexes included. Every tag of the repository is in the tree, or the one named. So rcp copy between two registries mirrors an image the way skopeo copy and crane copy do; a copy from a registry to a directory writes a layout skopeo’s oci: transport reads (and byte for byte what it would write, index.json included); a copy from a layout pushes it; verify compares two registries or a layout against one (-nomods, since a registry keeps no times, modes or owners); scan lists what an image is made of. Names are content, so a blob the destination already holds is neither uploaded nor read from the source, and a second mirror of the same image moves nothing but its manifests. A blob skipped that way is recorded in the catalog with an empty digest column, as a kernel copy is: it was never read, and the registry’s own content-addressing is what proved it holds those exact bytes. The skip is a HEAD on the destination repository and nothing more, so it is exactly as good as the registry’s promise that a digest names its bytes; rcp does not re-read to check, and a registry that answers HEAD for content it does not have would leave the manifest unpushable. (The stand-in keeps one blob store for every repository, which is why the second-mirror measurement above moves no blob; a registry that links blobs per repository would answer HEAD with a 404 there and take the upload.) Uploads stream through the chunked upload protocol; manifests are staged as they arrive and pushed, children before parents and tagged as index.json says, when the copy finishes, because a registry refuses a manifest whose blobs it does not have yet. Nothing but the sniff can tell a manifest from a layer that happens to be JSON, so what index.json reaches decides: what it does not reach goes up as the blob it is. A layout with no index.json reaches nothing, and that used to be a silent success — every manifest went up as a blob no client can name, no tag was pushed, and the run said PASSED, measured at 3 blobs and 0 manifests. oci-layout is the source saying it is a layout, and a layout owes an index, so a run that wrote one and got an index naming nothing now fails, saying the copy would create no image.

Be exact about what that prevents, because an earlier sentence here was not: it stops the manifests, which are held back until sync, and it cannot unsend a plain blob, which streams up as the copy reaches it. On the measured layout that is 1 upload where the old code sent 3. Unreferenced blobs are what a registry’s garbage collection is for; an image that does not exist is not. Downloads are digested as they pass and a blob whose bytes do not match its name is refused. docker.io is Docker Hub (library/ is added for a bare name). Authentication is the registry’s own: a Bearer challenge is answered from the realm it names, a Basic one with the credentials; those come from -oci.user USER:PASSWORD, RCP_OCI_USER with RCP_OCI_PASSWORD, or the container tools’ own files (REGISTRY_AUTH_FILE, ~/.docker/config.json, $XDG_RUNTIME_DIR/containers/auth.json). -oci.insecure uses plain HTTP, -oci.noverify skips certificate checks; HTTPS needs a build with the rustls feature, as S3 does. What a registry has no room for is refused in the open: a path that is not a blob, a symlink, a rename. delete removes tags and manifests; blobs stay until the registry’s own garbage collection, which is the registry’s job.

A stream that dies mid-blob is reissued from where it stopped. A token expires with the body already flowing – Docker Hub’s lasts about five minutes and a large image outlives it – and there is no 401 left to answer: the 200 and its headers are spent, so the registry just stops sending. The retry before the first byte cannot reach that. rcp reissues the GET with Range: bytes=N- from the byte it stopped at, which is where a 401 can be answered, and carries the running digest across, because a range resume skips nothing and that digest is the only thing making a partial read safe. Bounded by progress: a resume is attempted once at a given offset, so one that moves no bytes is the last, and 64 consecutive reconnections that each deliver under 1024 bytes stop a server that dribbles and hangs up. A reconnection that delivers a kilobyte or more resets that count. And a Range is a request, not a guarantee – a server free to ignore it answers 200 with the whole blob, which would put the blob’s head at its tail – so a ranged read that is not answered 206 is refused before a byte lands at the wrong offset, naming the server rather than the blob. The digest would catch it on this path, since the hasher rides across; it would not on the Seek path, which carries no hasher, and the refusal covers both. The executable control for the reissue is a stand-in that cuts and then refuses to honor the Range: with the resume load-bearing, that run must fail, and does.

An upload goes where the registry sends it. POST .../blobs/uploads/ answers with a Location, and the spec puts no constraint on where it points — ECR and GAR hand the body to a signed URL on their own storage host. rcp follows it, on a connection to that host, and deliberately sends it no Authorization: the signature in the URL is the credential, and this registry’s token is not for a third party to see. A relative Location from that host means a path on that host, not back on the registry. It is tempting to read only the path; that sends the body to the original registry under a path that means nothing there — and against a stand-in that serves both, the copy succeeded while never touching the host it was sent to. The contract arm runs a stand-in that hands uploads to a second listener, requires the blobs to arrive there, and requires the credential not to.

The bearer realm is the sharper half of the same thing. A 401 names a realm to fetch a token from, and fetching one sends the basic credentials — so a registry answering WWW-Authenticate: Bearer realm="http://elsewhere/token" is handed this run’s username and password. Measured before the gate existed: a run railed to -allow oci://HOST/app sent Basic b3BlcmF0b3I6c3VwZXJzZWNyZXQ= to a listener on another port. Docker Hub’s realm really is another host, so following it is the feature; what bounds it is that a railed run follows an upload Location or a realm only to an authority its own -allow roots name. hdfs:// is the third protocol with this shape — a namenode’s 307 chooses the datanode that receives the file body — and it is gated by the same rule, which now lives once in ports rather than inside the registry client.

A -nodes fleet cannot have a registry as its destination, and says so. Each worker is its own process holding its own staged manifests, and only this process calls sync, so the manifests were never pushed: measured, copy -nodes exited 0 reporting 8 files while the registry took 3 blobs, 0 manifests, and had no such repository at all. A silent success that produces nothing pullable is worse than a refusal, so the pair is refused — by the coordinator and by copybatch on the worker side. Blob uploads are already eight in flight within one process.

Archives: tar://

tar://ARCHIVE[!/SUBPATH] is a tar archive as a tree. rcp copy DIR tar://out.tar packs; rcp copy tar://out.tar DIR unpacks; rcp scan tar://out.tar says what is in one without extracting it, and !/sub roots at a directory inside. Reading opens a member by seeking to its offset, because the archive is scanned once and the offsets kept — so unpacking one file out of a hundred gigabytes reads that file, not everything before it.

-rsyncable cuts the .tar.gz members where the CONTENT says to instead of every 4 MiB, which is pigz --rsyncable reached through this crate’s own Gear chunker. The members were already independent — that is what gives parallel decode, and it is pigz -i — but their boundaries were at fixed input offsets, so inserting one byte near the front shifted every later boundary and rewrote the whole tail. Measured through the CLI on 12 MB of log-like text with a 25-byte insertion at offset 3000: fixed cuts keep 0 of 3 members, -rsyncable keeps 25 of 27. gzip -t and tar tzf accept both, because the format did not change — only where the cuts fall. It costs about 0.5% of ratio, and an arm fails if that grows past 5%.

The format written is POSIX ustar with pax extended headers where ustar has no room: a path over 100 bytes that will not split at a /, a size over 8 GiB, a link target over 100 bytes, a time with nanoseconds, an id over 2097151. Reading also takes GNU’s L and K long-name members, which is what tar cf writes by default, and GNU’s base-256 numbers. Hard links are members of type 1 and come back as one inode on the other side; symlinks are type 2.

The interoperability claim is measured in both directions, because an archive only another rcp can read is not an archive: a contract arm packs a tree, has tar xf extract it and compares, then has tar cf write one (with names too long for ustar) and reads it back with rcp. Both comparisons are rcp verify, which checks the bytes and the metadata rather than the listing.

A .tar.gz is read too; without a decoder it SILENTLY COPIES NOTHING. Measured without one:

$ rcp copy -noId tar:///tmp/a.tar.gz /dest
copy: 0 files (0 bytes), 0 skipped, 1 directories, ...
STATUS: PASSED

The archive held two files. A gzip member’s first 512 bytes are not a tar header, so the scan found no members, and an archive with no members is not an error — the run reported success and wrote nothing, which is the worst shape a data mover has because the operator has no reason to look. deflate.rs is DEFLATE (RFC 1951) and the gzip container (RFC 1952) in the standard library alone, and the archive is decompressed once to a temp file rather than into memory, so the seek-to-a-member’s-offset design is kept. The decode still holds the compressed input and the expanded archive in memory before that temp file is written, and MAX_EXPANDED is that memory ceiling; a multi-gigabyte archive is not streamed to disk yet. The CRC-32 and length in each member’s trailer are checked and there is no flag to skip them: they are the only thing separating a correct decode from a plausible one. A multi-member stream — cat a.gz b.gz, or what pigz -i writes — decodes as the whole thing rather than its first member.

Writing one is supported too. Falling back to a plain tar for a tar:// destination whose name ends in .gz would be a defect even if documented: the run would say PASSED and leave a file that tar xzf and gzip -d both reject. A limitation written down in a README does not reach the operator who is reading STATUS: PASSED.

The encoder is DEFLATE with a 32 KiB window and a bounded hash chain, in deflate.rs beside the decoder. Three things about it are load-bearing:

• Each block is written stored, fixed-Huffman or dynamic-Huffman, whichever counts fewest bits — an exact count of what each would emit, header included, not a heuristic. The two outer choices are there for opposite reasons. Stored is the floor: with fixed Huffman alone a literal costs 8 or 9 bits, so uniformly random bytes come out at 8.4375 bits each, measured at +5.44% on 200 KB of noise — rcp copy /tree tar://out.tar.gz over an OCI layout or a photo library is exactly that input, and the .tar.gz came back larger than the .tar under a name that promises the opposite. A stored block costs five bytes per 64 KiB. Dynamic is the ceiling: a code fitted to the block’s own histogram, with its lengths bounded at 15 bits by package-merge rather than by perturbing the frequencies until a plain tree happens to fit. On the 400 KB 4-symbol corpus it takes the output from 179,942 to 120,430 (0.450 → 0.301), and on 1,029,896 bytes of this crate’s source and prose from 377,398 to 320,057 (0.366 → 0.311).

• The matcher is lazy, not greedy. Having found a match it holds it, looks one byte further, and takes the later match instead if that one is strictly longer — buying a literal to cover more bytes. > and not >=: an equal-length match one position later is never better and usually sits at a greater distance. It costs a chain walk per byte rather than per match, bounded by taking any match of 32 bytes or more on the spot. Worth 2.7% on source and prose (320,057 → 311,493) and 0.2% on the pathological 4-symbol corpus — and the symbol count goes UP while doing it, from 73,851 to 85,091, because spending literals is the mechanism rather than a side effect.

• Members compress in parallel, and the bytes do not change. Because members are independent and each is a pure function of its input, moving the work to a pool alters only which thread computes it — -parallel sets the worker count, an ordered queue restores submission order on the writing side, and at most one block per worker is in flight so memory stays bounded at roughly 4 MiB each. On 16 MB of source and prose across 4 cores: 1.27 s → 0.38 s at 4 workers (3.3×), output byte-identical at 1, 2, 4 and 8 workers. Eight workers buy little here because that input is only four members.

• The stream is a run of independent gzip members, one per 4 MiB of input, which is what pigz -i writes and what RFC 1952 §2.2 allows. deflate() compresses a whole slice, so wrapping the archive in one member would mean holding the archive in memory — the wrong shape for a tree of any size. Each member starts with an empty match window, which costs a fraction of a percent at 4 MiB and is exactly the property that lets a reader decode members in parallel.

• Reading sniffs the magic bytes; writing reads the name. A file that does not exist yet has no bytes to sniff, so on create the name is binding — and a name saying a wrapper rcp does not write (.bz2, .xz, .lz, …) is refused before the file is made, rather than quietly given a plain tar. That is the same defect under a different suffix.

.tar.zst is read and written the same way, through the crate’s own zstd codec (the one -compress.codec zstd uses on the wire). A name ending in .zst or .tzst writes one zstd frame per 4 MiB of input, which is the zstd equivalent of the gzip members above: concatenated frames are one file by definition (RFC 8878 §3.1), -parallel compresses them on a pool with byte-identical output, and -rsyncable cuts them where the content says to. Reading sniffs the frame magic and also a leading skippable frame, which is how pzstd begins every archive. A contract arm reads archives made by the reference zstd -19 and by pzstd, and has the reference zstd -t and tar --zstd -xf read what rcp wrote.

The ratio is gzip’s, not zstd’s, and the numbers say so. On 3.3 MB of this crate’s source and docs: rcp’s .tar.zst is 991,256 bytes, rcp’s .tar.gz is 979,640, the reference zstd -3 is 922,291 and zstd -19 is 705,977. The encoder has one greedy match search and 4 MiB frames with no shared history; what .tar.zst buys today is the format, for a consumer that wants zstd, and a decode that is not slower than gzip’s. Before this, a .tar.zst source was refused as “not a tar archive” and a .tar.zst destination was refused by name.

The correctness claim is against the reference implementation, not a round trip: the_system_gzip_reads_what_this_encoder_wrote puts rcp’s output through the system gzip -d, gzip -dN and tar xzf, and a contract arm packs a real tree to tar://out.tar.gz and has tar -xzf unpack it. An encoder and a decoder that share a misunderstanding agree with each other perfectly, so rcp reading back its own archive would have proven nothing — it sniffs the magic and falls through to a plain tar. The shapes deliberately include a run long enough to force length 258 (which RFC 1951 lets you encode two ways, and rcp’s own decoder accepts both) and a match at exactly the 32 KiB window edge (distance 32768, the largest legal one), because those are the cases where picking the wrong code still round-trips locally.

How far it is from gzip, measured so the attribution is not guesswork. Comparing against gzip -6 cannot say whether the difference is the matcher or the code table, because gzip changes both at once. zlib’s Z_FIXED separates them — on a 400 KB 4-symbol corpus, Z_FIXED level 1 is 0.533, Z_FIXED level 9 is 0.402, and the default level 9 is 0.288. So zlib’s strongest matcher, restricted to fixed blocks, still only reaches 0.402 — the code table was doing most of the work, which is why dynamic blocks were the thing worth building and no chain depth substituted for them.

Both halves of that are now done, and the second one barely moves this corpus while mattering on real input — worth stating, because the 400 KB fixture is deliberately pathological and is not the thing anyone compresses:

Note 0.402 was never a floor for fixed-Huffman encoders — it is one strong matcher’s result, and an optimal parser (zopfli, libdeflate) emitting fixed blocks could go lower. The remaining gap is an optimal parse, which rcp does not attempt.

What a stream has no room for is refused in the open. There is no rename, no removal and no editing in place: delete on an archive is refused rather than half-done. An archive that already has something in it is never rewritten — writing one truncates from the start, and a truncated destination lists as empty, so no plan could show the loss and -requireconfirm could not catch it; the refusal is the only honest answer. A tar destination is a single stream, so its files are written one at a time whatever -parallel says: measured before that was enforced, a copy with the default worker count produced an archive GNU tar read as far as the first interleaved member and then reported “Skipping to next header” for the rest. And the archive is not a tar until the two zero blocks at the end are written, which is sync’s job — so a tar destination is one of the two whose sync publishes, beside a registry, and a fleet cannot write one.

Holes ARE carried, as GNU pax 1.0 sparse members — a GNU.sparse.* extended header naming the real file and its real size, and a decimal extent map at the front of the member’s data. tar --sparse --format=posix writes the same shape, and reads this one: a 64 MiB file holding 128 real bytes packs to 9,216 bytes rather than 67,113,472, and tar xf extracts it sparse.

The limit is where the decision has to be made. A member’s header precedes its bytes and a sparse header needs the whole extent map, so the writer holds the first 8 MiB waiting to see a hole; a file solid for longer than that is written flat even if a hole comes later — correct, and larger.

Reading covers all four encodings GNU tar can write, and it has to: only 1.0 carries GNU.sparse.major, so a reader that gates on that key treats 0.0 and 0.1 as ordinary members, reads the extents as the whole file, and reports success on a file that is not the source.

Extended attributes are not written (pax can carry SCHILY.xattr.*; this build does not), and a run that asks for them is told.

Command-line reference

The same tables are available at run time, as JSON, from rcp describe.

Commands

Endpoint schemes

Options

Events

With -json, each line on stdout is one of these events.

Exit codes

Design

A hexagon, all the way through. ports.rs holds every interface the core talks through: Tree and Sink for a storage location (list, read as segments of data and holes, create_file, kernel_copy, metadata, attributes), Digests for the content hash, Throttle for the read rate, Log for debug lines, Names for user and group ids, and Batches for a worker fleet; Runtime bundles the cross-cutting ones. The core (plan.rs, transfer.rs, scan.rs, catalog.rs, filter.rs) is written against those ports only and keeps no process-wide state: plan.rs turns listings into actions with no I/O, so the resume, sync and hard-link rules are unit-tested in microseconds, and transfer.rs and scan.rs run the use cases. The driven adapters are LocalTree (with uring.rs as its fast engine), NfsTree and Nfs4Tree, S3Tree, HdfsTree, the two digests, the token bucket, the stderr log, the system name tables and the ssh fleet, each taking its settings at construction; anytree.rs is the static dispatch over the trees and Transports the settings of every remote kind. The driving adapter is the CLI in main.rs: it parses the options, assembles a Runtime and Transports, opens the trees and calls the use cases; a library caller starts from Runtime::plain(). tests/hexagon.rs reads the source and fails when a core module names an adapter or keeps a static, with a control that plants an offense and checks the scan sees it. unsafe is the syscalls a copy needs – tree, uring, gss, xattr – plus seven libc declarations in five other modules: setsid and kill (jobs), signal (cluster and main), prctl and getppid (mcp), isatty (lib). Hand-kept versions of this sentence go wrong easily: naming only the four modules, miscounting the declarations, or attributing a fork to cluster, which this crate does not call anywhere. The list above came from grep -rn 'extern "C"' src/, and so should the next one: a hand-kept inventory of the code is the code’s second copy, and the second copy is the one that rots. The protocol code (nfs/: XDR, SUN RPC, NFSv3; s3/: HTTP, SigV4, the XML that the service speaks) is written against the standard library and tested against in-process stand-in servers that speak the real wire formats.

Design notes

This document traces rcp’s design to what the literature and comparable tools actually do, records what rcp implements and plans, and gives the reasoning, measurements and rejected alternatives behind each design decision. The literature survey is dated 2026-09-21. Every survey entry names its source; where a claim could not be verified from the source itself it says so.

Literature survey

Content-defined chunking and delta synchronization

• FastCDC (Xia et al., USENIX ATC 2016; extended in IEEE TPDS 2020) replaced Rabin fingerprinting with a Gear rolling hash, cut-point skipping below a minimum size, and “normalized chunking” to narrow the chunk-size distribution. About 10x the throughput of Rabin-based chunking at nearly the same deduplication ratio. usenix.org/conference/atc16/technical-sessions/presentation/xia
• A Thorough Investigation of Content-Defined Chunking Algorithms (arXiv 2409.06066, v3 August 2026) compared the leading chunkers on throughput, deduplication ratio, average chunk size and variance. It recommends Gear with normalized chunking as the practical choice (window 32 bytes, bit threshold log2(target - window), NC-1..NC-3), and finds that normalization reduces variance with only marginal loss of deduplication. arxiv.org/abs/2409.06066
• VectorCDC (arXiv 2508.05797, v2 January 2026) accelerates hashless chunkers with SIMD; 8-26x over previous vector-accelerated chunkers. Relevant as the ceiling, not as a requirement: Gear at ~600 MiB/s per core is about 4.8 Gbit/s, ahead of the 1 and 10 Gbit paths rcp usually moves data over and behind a 25 or 100 Gbit one, where chunking would want either VectorCDC or a core per stream. arxiv.org/abs/2508.05797
• Dsync (Xiao et al.) and NetSync (IEEE TPDS 2022) redesign rsync’s delta protocol around CDC: the weak hash FastFp is the sum of the Gear rolling hashes produced while chunking, so it costs nothing extra; strong hashes are computed only for weak-hash matches, and the protocol exploits locality. Both are several times faster than rsync’s byte-sliding window at similar bandwidth savings. lingfenghsiang.github.io/docs/DSync.pdf, NetSync (TPDS’22)
• SkySync (USENIX FAST 2026) uses the Dsync workflow as its baseline and describes it step by step: client chunks and sends weak checksums with chunk metadata; the server chunks its copy, matches weak checksums, computes strong checksums for matches only and returns them; the client confirms with its own strong checksums. usenix.org FAST’26 prepub
• Chunking attacks on backup services (arXiv 2504.02095, April 2025) show that CDC parameters leak through chunk sizes when an attacker can observe them. For rcp the chunk boundaries never leave an authenticated ssh channel, and the Gear table is not a secret; noted so nobody later treats the chunker as a confidentiality mechanism. arxiv.org/pdf/2504.02095

Online tuning of parallelism

• Marlin (“Use Only What You Need: Judicious Parallelism for File Transfers in High Performance Networks”, Arifuzzaman and Arslan, ACM ICS 2023) separates read, network and write parallelism and tunes each with online gradient descent, on the observation that one parallelism level for all three overburdens the ones that need less. dl.acm.org/doi/10.1145/3577193.3593722
• FastBioDL (arXiv 2508.05511, August 2025) frames a parallel downloader as online optimization: a utility function of throughput and stream count, gradient descent on the number of concurrent streams, up to 4x over static tools. arxiv.org/abs/2508.05511
• AutoMDT (arXiv 2508.05546, SC’25 workshops) and Elastic Data Transfer Optimization with Hybrid Reinforcement Learning (arXiv 2511.06159) continue the line; Globus’s static concurrency loses to the online tuners in every table shown. arxiv.org/abs/2508.05546, arxiv.org/abs/2511.06159

Integrity hashing

• TeraCopy’s authors recommend xxHash3 or BLAKE3 for copy verification and MD5 or CRC32 only for interchange; BLAKE3 runs at 4-7 GB/s per core on recent x86-64, SHA-256 with SHA-NI at 1.5-2.5 GB/s, portable SHA-256 at 300-500 MB/s. codesector.com/how-to/choose-checksum-file-format
• FIPS 140-3 approves only SHA-2 and SHA-3; BLAKE3 is not on the list. So BLAKE3 is the fast option and SHA-256 stays the default where a published spec names it. That is a statement about the algorithm: rcp’s SHA-256 is its own code, not a validated module, and a deployment that needs a validated module needs one from outside this crate. devtoolspro.org SHA-256 alternatives 2026

What comparable tools ship

• rsync 3.5.0 (August 2026) is a security release (33 fixes); its data path is unchanged since 3.2: xxh3/xxh128 checksums, zstd and lz4 compression, byte-sliding delta transfer. No copy_file_range, no CDC. rsync NEWS
• rclone 1.75.1 (September 2026): dozens of cloud backends, multi-thread transfers for capable backends, a hasher overlay caching checksums, an sftp host-key pinning flag. Go, no delta transfer. rclone changelog
• wcp (io_uring file copy) measured 2.6 GB/s against cp at 1.2 GB/s and rsync at 0.6 GB/s locally, at the cost of portability and memory. github.com/wheybags/wcp
• QUIC: ESnet’s guidance is that transfer tools have not adopted it, that the HTTP/3 stacks they measured run 2-5x slower than HTTP/2 on 10G links because QUIC lives in user space, and that HPN-SSH is the practical answer on high-RTT paths. Two qualifications: those are measurements of particular implementations at one link speed, not a property of the protocol; and HPN-SSH is a patched OpenSSH at both ends, so it is an option a site can choose, not something rcp can assume. What rcp does about high-RTT paths on stock OpenSSH is open several sessions (one per worker) so the aggregate is not bound by one channel’s window. The QUIC transfer tools on GitHub are small peer-to-peer projects (quic-send, quicfiletransfer). fasterdata.es.net/data-transfer-tools/quic

What rcp does

The base: io_uring on the local path, copy_file_range (a reflink where the filesystem offers one), sparse files by extent and by zero detection, pipelined NFS v3 and v4.0 with Kerberos, S3 and HDFS clients with no SDK, SHA-256 verification with SHA-NI, a catalog for resume and sync, and -nodes scale-out over ssh. Measured against the literature, that base lacked a fast non-FIPS hash, a delta protocol, an rcp-to-rcp transport that a delta protocol can run on, rate limiting, and any online tuning of parallelism. The roadmap below closes those gaps and adds endpoints and an agent interface.

Roadmap

Each item names the test that decides it is done.

Agent interface

A copy tool an agent can drive needs, in this order: output it can parse, a description it can discover, bounds it cannot argue past, and a way to let go of a long job.

Delta sync

The -delta design, decision by decision:

• Direction, and placement. Destination signatures travel to the source; matching bytes are copied at the destination. That is Dsync’s shape and the right one for a copy tool. The matching has to run where the source bytes are, as an operation on the source tree (delta(rel, signatures) -> instructions): the local adapter matches in process, the ssh adapter sends the signatures to its serve and gets the instruction stream back. Matching in the core would pull every source byte across the wire for an ssh:// source and save nothing on the path that matters. Between two ssh endpoints the client is a relay, as the README says.
• Two tiers. A single strong hash per chunk on both sides looks cheaper than a second tier when BLAKE3 runs at several GB/s. Measured, it is not: an 8 KiB chunk is far too small for BLAKE3’s tree parallelism, so the per-chunk pass ran at 266 MiB/s against 1179 MiB/s for the whole file, and was 82% of the source side’s hashing (50% under SHA-256, 38% under xxh3). The Gear fingerprint the chunker already computes is free, so it filters and the strong digest decides, as Dsync describes; a source chunk whose fingerprint no signature carries never has its strong digest computed. The weak tier adds eight bytes per signature on the wire, 0.1% of the file. It is protocol VERSION 2, unconditionally, because the serve refuses a version it does not speak, by name.
• Chunk size scales with file size, the way rsync’s block size does, so a terabyte file does not ship a gigabyte of signatures. The average chunk follows the square root of the file (rsync’s rule) rather than a fixed divisor, so a terabyte file has a million chunks and a petabyte one 256 million rather than a billion.
• Patch to a temporary file, then rename. In-place patching corrupts when a chunk is read from an offset that an earlier instruction already overwrote, and it rewrites every other hard link of the destination inode. The temporary file is also what makes an interrupted patch leave the old file intact. It is created exclusively under an unpredictable name (a planted symlink or a peer on a shared export with the same pid gets a refusal, not a shared write), with mode 0600, and the directory is synced after the rename when -fsync asks.
• Basis identity and a final digest. The signatures name the basis (size, mtime, inode). The basis is signed, then opened and checked, so the identity check covers the signing window; the patch refuses a basis that changed after signing, and the reconstructed file’s whole-file digest is checked against the source’s before the rename.
• Shorter source. The instruction stream ends with the final length; a destination longer than the source is truncated by construction, never by omission.
• Fallback. A patch that cannot proceed for any reason falls back to a whole copy.
• Holes. Holes are chunked as zeros on both ends and the zero chunks go out as holes, so a sparse file matches itself.
• Bounds. A signature is 44 bytes inline (a million of them, the most a terabyte file produces at the largest chunk, is about 44 MiB per side plus the map). A serve refuses more than eight million signatures for one file and more than sixty-four setups per session.

Weak hash and streamed signatures

• The fingerprint is free. Chunk.weak is the running sum of the Gear hashes the chunker computes anyway, which is exactly Dsync’s FastFp. No second pass.
• The strong digest is the second tier. Matcher indexes signatures by (weak) and computes a per-chunk digest only when a fingerprint hits. The module header carries the measurement above that motivates it.
• Signatures stream. SIG_BATCH frames of LIST_BATCH = 2048 signatures under one sequence number, like listings, terminated by SIG_END, with MAX_SIGNATURES = 8M bounding one delta setup at about 350 MiB.

A wire-byte criterion cannot measure the batching; the contract suite does. “Signature bytes on the wire drop about 8x” is a statement about the weak tier; a build that sent only the first batch would satisfy it and silently lose every chunk past the 2048th. What holds the line is tests/contract.rs: a 32 MiB fixture chunks at an 8 KiB average into ~4096 chunks, which is two batches, and the test asserts a 1 MiB edit moves between 1 MiB and 1 MiB + 128 KiB of literals. Drop the second batch and the literal count jumps by megabytes. The multi-batch path is not merely executed there — it is load-bearing for an assertion.

The index cannot stream, and that is not a gap. Tree::signatures returns a Vec and Matcher::new takes a slice, so the list is materialized on both sides. That is inherent: a matcher is a lookup table, and you cannot index against signatures you have thrown away. Transmission streams; the index does not, and the MAX_SIGNATURES cap is what bounds it.

Not yet placed, and deliberately not planned

Not yet placed: several streams per large file (the -parallel unit is a file); concurrent ranged GETs and multipart PUTs on S3; resuming a partly written object rather than only a partly copied tree; and ControlMaster, which is ssh configuration rather than rcp’s to add but is what makes one session per worker cheap.

The Gear table here comes from a seeded splitmix64, not the table published with FastCDC, so the paper’s deduplication ratios are a guide, not a measurement of this chunker; the protocol is closed at both ends, so nothing depends on the table being the published one.

Deliberately not planned: QUIC (see ESnet above; rcp’s remote transport runs inside ssh, which is where the field’s high-RTT answer, HPN-SSH, lives), and NFSv4.1 sessions (no data-path gain over 4.0 for a copy tool; 4.2’s server-side COPY is the one worth having and needs 4.1 first, so it stays a later item).

Safety rails, the MCP server and the registry endpoint

Every issue below was reproduced before anything was changed, and each fix has a contract arm with a control that fails when the fix is removed. Suspected defects that measurement showed not to exist are recorded too, with the reason.

Safety rails

Open: -allow is a check before the open, so it cannot close a TOCTOU race.

MCP server

Relaying a credential on the command line cannot work. rails() always injects -requireconfirm, which makes the MCP child a railed run; a railed run refuses credentials on its command line; so a server started as rcp mcp -oci.user U:P that relays -oci.user on the child’s command line refuses its own credential. A unit test that reads rails() output and never runs a child cannot see this. Credentials and transport configuration are relayed through the child’s ENVIRONMENT, which is the split the rails already rest on, and the contract suite starts a real server against a registry that demands a credential.

The plan digest is reproducibility, not approval. It proves the plan a run would carry out is the plan that was printed, and nothing about who read it. It is not a substitute for an approval that binds a person, a session and a time, and the README should not be read as claiming one.

Open: the child’s output is held whole in memory, so a run with millions of entries has no back pressure and no progress until it ends.

Registry credentials and hand-offs

A registry names where the credentials go. A 401 carries WWW-Authenticate: Bearer realm="...", the client fetches a token from it, and it sends the BASIC CREDENTIALS to get one. Docker Hub’s realm really is another host, so this is the protocol working – and it means a registry that answers with a realm of its choosing is handed the run’s username and password. Measured from a run railed to the registry alone: Basic b3BlcmF0b3I6c3VwZXJzZWNyZXQ= arrived at a listener on another port. The upload hand-off moves blob BYTES to another host; the realm moves the credential. Both go through one policy.

Open: a token that expires mid-body on a multi-gigabyte blob GET is not renewed (the retry is before the first byte); and stripping Authorization on every away host is right for signed URLs and wrong for a storage host under the same auth realm, and nothing here can measure which a real registry wants.

Registry publishing and reads

A resume after a failed publish does not skip the manifests it owes. Mode::Resume skips a file only when the catalog knows it and the destination already shows it, and a registry with no tags shows nothing, so the resume re-reads index.json, re-stages and publishes. A contract arm holds that end to end. A committed flag to keep the catalog from recording a staged manifest as done was tried and removed: impl Sink for AnySink had no committed delegation, so the default always won, and a successful publish reported all 8 entry events and 11 catalog rows. Wiring it up would have introduced the failure it was meant to prevent. The conjunctive skip is therefore what stands between a catalog record and a skipped manifest, and any change that lets resume trust the catalog alone would remove it.

Two cases stay open with reasons rather than fixes. A source with no oci-layout whose blobs sniff as manifests still publishes nothing and passes – that is the same rule that keeps a deliberate blob-only push working, and a source with no marker is not claiming to be a layout. And a filesystem sync failure has no arm: making syncfs fail needs either a fault-injection hook in production code or a stub Tree implementing 23 methods, neither worth it to guard a three-line change. Content-Range is not yet sent on a PATCH, which some registries require.

Digest cache

The catalog records which algorithm produced its digests. Its digest column holds hex, and hex says nothing about what produced it. With one algorithm that is invisible; with four it becomes a mixture — measured:

$ rcp copy -newid m -blake3 src dst
$ rcp sync -id m                       # the default, sha256
a.txt e0e63aa4...   (BLAKE3)
b.txt 7f8b1dfc...   (SHA-256)

Same column, same file, nothing to tell them apart, both runs PASSED, and scan -edupe groups files by that column. The catalog records its algorithm and refuses a continuation under a different one, naming the flag that would work. That is a prerequisite for a cache rather than a cache: there is no trusting a stored digest until you know what it is a digest in.

A verify that answers from a cache is not a verify. It reports on what was read last time, keyed on metadata the thing being checked also controls. The failure it would miss is the one verify exists for: content changed while size and mtime were preserved, which is a restore from a bad backup, a filesystem that lost a write, and a deliberate edit, all at once.

So the shape is narrower than rclone’s, because rclone’s hasher overlay sits in front of a REMOTE whose bytes cost money and time to re-read, and rcp’s verify is usually reading local disk on both sides:

• scan may use it. A listing is not a check, and re-reading a tree to print its digests is the case the cache is for.
• verify may not, on either side, unless the operator asks for it in those words — a -trustcache that says plainly it is comparing records rather than bytes. That flag does not exist.
• copy and sync gain nothing: they read the file in order to write it, so the digest is free.

What is built is exactly that: scan -hashcache, and nothing else. verify, copy and sync do not consult it and no flag makes them.

The cache lives beside the catalogs, one file per (root, algorithm), and is keyed on size, nanosecond mtime and inode. Measured on 128 files / 1.07 GiB of random data, four cores, SHA-256, with a warm page cache – the condition least favorable to a cache, since a cold tree would make the first column slower still:

The digests are identical to the uncached scan and to sha256sum, checked by diff. The file is 14 KB for 128 entries, mode 0600.

The number that carries the claim is the right-hand column, not the left. “The second run was faster” is satisfied by a warm page cache doing nothing at all, so scan counts the files it hands to its workers and the arms assert on that count – zero with the cache, 128 with it removed as the control.

Three decisions, each a defect avoided rather than a preference:

• ino earns its place only on a local tree. Every adapter without a real inode synthesizes one as fnv64(path) – stable across runs by design, which is right for sync’s move detection and worthless as a change detector, since a function of the key cannot disagree with itself. There the check is size and mtime, exactly rclone’s. And oci:// reports mtime_ns: 0 for every blob, which would collapse it to size alone, so a zero mtime is refused a hit and not stored.
• A cache is never allowed to fail a scan. Truncated, malformed, from another root, from another algorithm: every one of those is “no cached digests”, and a malformed line is skipped rather than half-parsed.
• Pruning is conditional on the run being complete. A filtered or depth-limited scan never looked at most of the tree, so evicting what it did not see would make the next full scan pay for this narrow one.

The limit is written down in the module and pinned by an arm that asserts the wrong answer on purpose: a writer that overwrites bytes in place without changing the length and then restores the mtime leaves all three fields identical, and -hashcache then reports a digest of content that is gone. Every mtime-keyed cache has this hole, rclone’s included. It is why the flag is opt-in: the same arm shows a scan without it still catching the tampering, so the operator who needs that answer has it by default.

Rsyncable gzip members

pigz has two related flags, and separating them is the whole insight. pigz -i writes independently-decodable blocks; this crate writes independent gzip MEMBERS, because concatenated members are a legal gzip file and that is what makes parallel decode possible. So -i comes free with the 4 MiB framing.

--rsyncable is the other half. With members cut at FIXED INPUT OFFSETS, inserting one byte near the front shifts every later boundary by one and every later member compresses differently. The archive is entirely new from the edit onward, and a delta transfer of the compressed file has nothing to match.

-rsyncable cuts members with the Gear chunker the crate already has for -delta. Measured end to end through the CLI on 12 MB of log-like text, inserting 25 bytes at offset 3000:

gzip -t and tar tzf accept both, which they must: the format does not change, only where the cuts fall.

The parameters are not pigz’s, and the reason is the unit. pigz cuts at roughly a 4 KiB average because its unit is a deflate block inside one stream, where a cut costs a flush. Here the unit is a gzip member — 18 bytes of frame, a fresh Huffman table and an empty match window — so a 4 KiB member would be mostly overhead. The average is a megabyte, with min = avg/4 and max = GZIP_BLOCK, which keeps the writer’s buffer bound exactly what it was.

What it costs. On compressible text: 1,018,028 bytes in 3 members with fixed cuts against 1,022,801 in 34 with content-defined ones — eleven times as many members for 0.47% more output. An arm fails if that grows past 5%. The member count is high there because Gear on low-entropy text cuts more often than the target; on incompressible noise it lands on 12 members for 12 MiB, which is the target exactly. Both numbers are in the arms rather than in this paragraph alone.

Test helpers should use the shipped decoder. A helper that splits a stream into members by decoding every prefix to find each boundary is quadratic, and on a 12 MiB fixture it does not finish. The shipped decoder’s gunzip_member already returns how many bytes it consumed. A test that reimplements the thing under test inherits both its cost and its bugs.

Three-way parallelism tuner

A single tuned number – how many workers are admitted to claim files – cannot say “read wider, write narrower”. Marlin’s argument is that a transfer has stages with different cost and different contention, so -parallel auto also tunes read and write concurrency, in src/stage.rs.

The decomposition is a pair of counted gates, not a pair of thread pools. A worker still owns one file end to end; it holds a READ permit while the source produces bytes and a WRITE permit while the sink takes them, and never both. So the read limit caps threads inside a read and the write limit caps threads inside a write, independently, with no channel between stages, no second buffer copy and no extra threads. The cost is that a worker re-queues for the read gate after each block, which is the mechanism rather than an overhead: it is what makes a read limit of two mean two threads reading.

Never both at once is load-bearing, so it has the sharpest arm in the module. A narrow read gate and a wide write gate: if the permit really is handed back, several threads write at once though only one may read, and the write gate’s peak occupancy exceeds one; if it were held across the write, that peak is pinned at one however wide the gate. Replanted – the whole defect is moving one take() after one enter() – it fails with “the write gate peaked at 1 with a read limit of 1”.

A contended sink, not a slow one, is the case for fewer writers. A merely slow sink wants MORE writers: if a write costs ten reads and the writes are independent, saturating the pipe needs ten writers per reader. The case where fewer is right is a contended sink – a device whose aggregate throughput peaks at a low queue depth and falls off past it, the ordinary behavior of a seeking disk or a rate-limited endpoint. That is the case a single worker count cannot express, and the one the arm models. Its control is the mirror image: move the contention to the read stage and the answer must swap, or the tuner is following the knob order rather than the landscape.

At most one knob changes per window. A single-knob loop measures a window, compares it with the previous one, and applies the verdict to the knob it is about to move – correct only when that is always the same knob. Carried to three knobs it credits each knob with the previous one’s effect. Remembering the move is not enough, because an undo is also a change: on “worse”, reversing and probing the next knob leaves two knobs different between consecutive windows and the next verdict is about both. Measured: with the undo folded into the same window, the write knob settles at 4 of a wanted 1, because each undo cancels the next knob’s move in the arithmetic. The undo gets its own window and is not itself judged.

The test stand-in must carry fractional bytes. A stand-in that advances progress.bytes by rate * tick truncates: a tick’s share of a rate of 500 is 0.5 bytes, and 0.5 as u64 is 0. Every configuration then feeds the tuner a flat zero, and every arm measures the tuner’s behavior on a dead stream. The stand-in carries the fraction, and scales the landscape to real byte rates so a 20 ms window is not pure quantization noise.

“The knobs move” and “the gates open” are different claims, and only the second is worth anything. The contract arm reads the tuner’s log, which reports a knob moving whether or not a single thread ever passes through the gate it sets. So Gate carries a cumulative admission count and a_staged_copy_takes_permits_and_a_kernel_copy_does_not runs a real copy_one against real local trees and reads it.

That arm also pins the boundary, which is wider than it first appears. A local copy with no digest wanted goes through copy_file_range, so the kernel does both halves and the stages are not entered: measured (0, 0) admissions, not (1, 0) – the file’s read permit is taken inside the branch that runs only when the kernel declines the copy. -hash is what declines it and puts the bytes on the metered path. So: the kernel-copy and io_uring paths are untuned because there is nothing there to separate, and the delta path has its own shape.

A fixed -parallel N carries no stages at all – not a gate wide enough not to bite, none – because a wide gate is still a mutex and a condvar on every block of every file, and a run that asked for a fixed worker count did not ask for a tuner’s machinery.

Tuner decision margin and noise

The attribution arm failed intermittently inside parallel suite runs – 4 readers (want 6) and 1 writers (want 1) – and passed 10 of 10 times alone. Its time budget was one cause (it is a window count, not a wall clock); the decision threshold was a second, independent one, and fixing the first does not fix the second.

The rig’s own noise is measurable. With both knobs held still, two consecutive windows of the stand-in differ by about 2% – and a fixed rule calling a window better above +3% and worse below -3% sits at roughly one standard deviation of the difference between two such windows, so a large share of its verdicts are coin flips. That is not a property of the rig: it is a property of comparing two single samples against a fixed threshold, and a real link is noisier than this stand-in, not quieter.

The clean demonstration needs no threshold at all. Give the tuner a landscape in which one knob appears NOWHERE in the rate. No setting of it is better or worse than any other, so any movement is wrong, and there is nothing to argue about. With the fixed threshold:

tune: writers 1 -> 2 flat, keeping, 304.4 MiB/s
tune: writers 2 -> 3 flat, keeping, 305.9 MiB/s
tune: writers 3 -> 4 worse,  undoing, 293.9 MiB/s
tune: writers 3 -> 2 better, keeping, 304.2 MiB/s

Two defects in four lines. flat, keeping is a ratchet – a knob that changes nothing reads flat whichever way it points, so it walks – and the worse/better verdicts are on 3.4% and 3.5% differences inside a 2% noise floor. Over one run the knob kept a move on 19 of 80 probes and visited all six of its values.

Three rules, each with its own replant:

The noise estimate costs nothing. Two of the windows the search already spends re-read a configuration it has just read: the window after an undo runs at the setting the probe was launched from, and the confirming window re-reads the probe’s own setting. Each is two readings of one configuration, so each difference is a noise sample by construction. No extra windows, no calibration phase beyond the two windows at the very start, and no constant to guess.

It rises fast and decays slowly, deliberately. When a machine gets busier the noise steps up at once, and a symmetric average spends the next several windows deciding by coin flip – which is the failure being guarded against, arriving exactly when it is most likely.

A known weakness: the margin’s replant fires only one time in three on this machine, because 2% noise makes a fixed 3% very nearly the right number by luck and the confirming window covers the rest. The margin is kept for the machine where that luck runs out. That is an argument about machines the arm has not run on, so it is stated as a reason and not as a result.

What it costs. A kept improvement takes three windows instead of two, so the search is about half as fast to climb; and a landscape with a flat SHELF before a rise is uncrossable, because a stalled knob probes one step at a time and undoes each. Both are accepted for the same reason: refusing to spend concurrency that no measurement asked for is the behavior wanted from a tuner, and a shelf is a speculative shape while the drift was measured.

Complexity arms count work instead of timing it

The same noise affects any arm that compares two wall-clock measurements. decoding_a_run_of_ampersands_is_linear_and_not_quadratic tells a linear decode from a quadratic one by timing both sizes, and failed under load with 8x the input cost 24.2x the time. Best-of-five runs does not help when the LARGE run is the one a loaded scheduler preempts in every sample; a bound that needs a quiet machine is not a bound.

So the instrument changed rather than its statistic: decode counts the bytes it looks at, in test builds only, and the arm compares COUNTS. Measured, and exactly, with no spread at all: 279,922 -> 2,239,922, a ratio of 8.0, against 200,030,000 -> 12,800,240,000, a ratio of 64.0 when the quadratic scan is replanted. A complexity class is a property of the work; timing is a proxy for the work that also measures the machine, and on a busy machine the machine wins. What it gives up is stated in the arm: wall-clock is no longer asserted, so a change that kept the scan linear while making each step expensive would pass.

Gear chunker performance

The delta path chunks every byte on both ends, so the chunker’s rate is a ceiling on what -delta can do however fast the disk and the network are. Measured: 664 MiB/s, 1.44 ns/byte on a 48 MiB buffer. Ten gigabit is 1192 MiB/s, so the chunker really is below that line.

The plan was lane-parallel Gear behind a runtime AVX2 check. What the measurement showed:

A fourth run on the same binary kept only its ratios: 0.99x for the lanes and 1.45x for the two-stage. It is reported separately rather than folded into the table, because per-variant numbers for that run were not recorded.

The lane row straddles 1.0. Four runs of the same binary on the same input, and one of them is a slowdown. On a four-core shared VM that is not a result, it is the noise floor – and it answers the question, because a SIMD kernel would have to clear that floor by a wide margin to be worth an intrinsic, a feature check and a block of unsafe. The reasoning behind the plan was that fp = (fp << 1) + GEAR[b] is a two-cycle dependency chain, so one stream cannot use a wide core; the measurement says something else limits this loop first. The lane code is kept – it is correct, it is proven equivalent, and it is the baseline any future SIMD attempt has to beat – but nothing calls it on the hot path, and no claim is made that it is faster.

What pays is separating the two things the loop does. The fused chunker tests the mask and tracks a length against min, avg and max, every byte. scan tests only the mask; Cuts::chunks then applies the length rules to one entry per candidate – 4.6 per chunk, about one per 2 KiB. Same output, 1.45x, stable across every run.

That separation is only legal because of two facts, and both are asserted rather than argued:

• The scan does not need to know where the cuts are. Bit j of the fingerprint depends on the last j + 1 bytes; the masks sit at bit 32 and above; min >= MIN_CHUNK is 256. So the reset Chunker performs at each cut is always further back than any mask bit can see, and the candidate positions are a property of the buffer alone.
• The weak hash survives the decoupling exactly. This is the part that could have killed it. Each chunk’s FastFp is the sum of fingerprints computed from a reset, which a continuous scan does not have. But the recurrence is linear mod 2^64, so running it from s instead of zero only adds s * 2^(k+1); summing over a chunk and using that 2^64 = 0, weak = sum(b) - sum(a-1) + 2 * fp(a-1). Exact for any chunk of 62 bytes or more, which every legal Params guarantees. Without it, cut positions could be found in parallel and every chunk would still need a second pass for its hash – the whole cost again. the_two_stage_scan_reproduces_the_chunker_exactly compares every field of every chunk across six inputs and three parameter sets, and its control drops the 2 * fp(a-1) term and requires the match to break.

Three pitfalls the measurement exposed. A perfectly periodic buffer yields zero candidates – the fingerprint enters a cycle that never meets the mask – so a lane-equivalence arm on it compares two empty lists and proves nothing; the non-vacuity check catches that. A constant buffer does the same, which is why every cut there is forced by max and lands where no candidate was recorded, and why the scan keeps a state rung every 4 KiB. And chunk_scanned deliberately does not emit the unfinished tail, so a rate test counting 5328 chunks against the fused chunker’s 5329 is an off-by-one in the test, not a chunker defect.

Not yet on the hot path, and the reason is specific. Chunker is a streaming API and the 1.45x comes from scanning a whole buffer at once, so wiring it in means buffering, which changes when Matcher emits patches and needs the under-62-byte final chunk handled separately. That is a change to the one piece of code both ends of every delta must agree on byte for byte. The decomposition is proven and measured; the wiring is the next step.

Socket transport (rcp://)

ssh:// pays for a child process and an ssh handshake per channel. For -nodes and -via across a tunnel somebody else already authenticated, that is all cost and no benefit, so rcp serve -listen ADDR speaks the same frames over TCP and rcp://host:port/path is the endpoint. Channel already abstracted over Box<dyn Read/Write + Send> with an Option<Child>, so the client side is a second constructor rather than a second transport.

What a socket does not inherit is the thing that matters. Over ssh://, OpenSSH has authenticated the peer and started rcp serve as that user before a frame is read; the protocol inherits an identity it never establishes. A socket inherits nothing, so served as-is on a routable address this would be an unauthenticated remote file service, read and write. The listener therefore refuses a non-loopback address unless mutual TLS authenticates the peer, and refuses to start at all without -allow. Refusing a capability is a smaller error than quietly shipping an open one.

The roots come from one place. The listener requires roots and the session enforces them, and those must be the same roots. With check_listen counting -allow while the session’s rail read only RCP_ALLOWED_ROOTS from the environment, serve -listen 127.0.0.1:P -allow /srv started happily, and a copy to /tmp/elsewhere reported PASSED with the directory created. Every refusal arm passed while that was true, and so did the happy path; the only thing that showed it was running the copy the rail was supposed to refuse. The roots are passed in rather than read from two places, and the contract arm replants the two-source version and requires it to fail.

Endpoint shape, and both parts are deliberate refusals rather than conveniences:

• No user. rcp://bob@host:port/path is refused. A username the protocol carries and nobody checks is an identity in appearance only, which is worse than none.
• No default port. ssh:// may omit 22; rcp:// may not omit its port. There is no registered port for this, and inventing one means a typo dials some other service.

Remote transport encryption (rcps://)

rcps:// is the rcp:// socket inside TLS. The obvious design is “add TLS, drop the loopback rule”. It is wrong: encryption is not authentication, and the rule follows authentication.

-tls.cert and -tls.key prove to the CLIENT who the serve is, and keep a passive observer out of the stream. They establish nothing about who connected. A serve with a certificate and no client verifier, bound to a routable address, is still an unauthenticated remote filesystem – now wearing a padlock, which is worse than the plain one because it reads as solved. So check_listen asks is_mutual(), which is exactly “was -tls.clientca named”, and a TLS serve without one is refused on a routable address precisely as a plaintext serve is. The contract arm measures that as a PAIR: the same wildcard bind with the same server certificate, refused without the client authority and accepted with it. Without the second half the arm would pass against a build that had stopped checking at all.

Two half-configurations are refused by name rather than resolved, because either resolution is a guess: a certificate without its key (-tls.cert alone), and a client authority with nothing to present (-tls.clientca alone, which asks the peer for a certificate while offering none and cannot complete a handshake).

The refusals are asserted by their own reasons. Probing the refusal cases with a RELATIVE source path makes every one of them “fail” on endpoint "src" is none of ... – before a socket is opened – and an assert!(!success) is satisfied by all four. What the arm asserts is what rustls actually says, and the three TLS cases are different by construction:

That last one is not a graceful message and it is not meant to be; what matters is that a plaintext client is not quietly served, and that the refusal is bounded rather than an allocation. The arm pins the number so that removing the limit shows up here.

One stream, two halves, and why the lock cannot deadlock. A TcpStream splits with try_clone because the kernel owns the two directions. A TLS connection cannot: the record layer, the sequence numbers and the key schedule are one piece of state. So the reader and the writer share one object behind a mutex. That is a deadlock in most shapes and is not one here: a Channel is a sequential request stream used by ONE worker, which writes its requests and then reads their replies on one thread, and a serve’s session is the same loop from the other side. Parallelism in this crate is more channels, never more users of one channel.

The -tls.* options are Transport, not Credential. Credential is what they look like, but a Credential is refused on any RAILED command line – and serve -listen is railed by construction, since it will not start without -allow. As Credential these would be options that could never be used on the only command that needs them (the contract arm reports -tls.cert chooses or carries a credential, which no rail can bound). Transport is the class for “how well is the connection to a named endpoint authenticated” and already holds -oci.insecure and -nfs.sec. It preserves the half that matters – a Transport is still refused from an MCP CALL, so a caller cannot name the trust anchor a railed server checks against, and relay_env still carries the operator’s choice to every child. What it gives up is value redaction in logs, and these values are file paths rather than keys.

Every scheme the parser accepts is described. describe and the endpoint ... is none of refusal a user reads after a typo are both checked against the parser. A guard that checks named adapters against their source files cannot see rcp://, which shares the ssh adapter and has no file of its own. every_scheme_the_parser_accepts_has_a_row derives the list from the parser’s own strip_prefix literals and checks both surfaces. Replanted, it names ["rcps", "rcp"] for each.

-via rcp:// needs an arm that actually dials the hop. connect_tcp_via is only reachable once -via accepts more than is_ssh(). An arm that copies to a LOCAL path through the hop passes without dialing it – -via routes the endpoints this process should not open itself, and a local path is not one. The arm therefore uses a second listener, because a hop’s rail bounds the endpoint TEXT: a root of /tmp/x does not admit rcp://host:port/tmp/x, the same rule the ssh relay arm spells ssh://filer/.... So the hop is started after the destination listener, with that listener’s port in its own -allow.

Not implemented: a serve that authenticates by anything other than a certificate – no token, no PSK, no unix-domain socket. None of those is needed while rcp:// on loopback plus a tunnel, or rcps:// with a client authority, covers the two cases that exist.

DEFLATE encoder

A tar:// destination named .tar.gz must contain gzip. Writing a plain tar under a .gz name and reporting PASSED is a defect, and a known-limitation note in the README does not fix it: nothing that prints STATUS: PASSED sends an operator to a paragraph in a README, and the file is discovered wherever it was shipped. That is why the crate has its own DEFLATE encoder.

The correctness claim is against the reference implementation, never a round trip. An encoder and a decoder that share a misunderstanding agree with each other perfectly, and rcp’s own reader is the worst possible judge here: it sniffs the gzip magic and falls through to a plain tar, so it reads a mislabeled plain tar happily. The arms put rcp’s bytes through the system gzip -d, gzip -dN and tar xzf, and a contract arm packs a real tree and has tar -xzf unpack it. Where gzip is absent the arm SKIPS and says what was not checked; a silent skip is a pass that checked nothing.

The hash chain, and why no arm saw its defect

The hash chain once broke in two ways at once — prev[i] left unset on the match path (truncating each bucket) and set to i itself on the literal path (a self-loop a later walk spins on). The stream stayed valid DEFLATE throughout, because every candidate is compared byte for byte before it is used, so the defect could only ever cost ratio.

Beware a fixture that cannot measure the thing. A corpus made of a 44-byte pangram repeated ten thousand times shows chain depth as irrelevant — a sweep from a 4-deep chain to a 128-deep one moves the ratio only 0.078 to 0.077 — but after the first sentence every position matches at distance 44, so chain depth cannot matter by construction. That is a measurement of the fixture, not of text, and its 0.008 ratio is the tell.

Measured on 1,029,896 bytes of this crate’s own source and prose, with both halves of the defect replanted into the real encoder and a narrow corpus beside it as a control:

The replant reproduces the defective encoder’s recorded 214,608 to the byte, which is what makes the prose row trustworthy. Against the system’s own encoder on the narrow corpus, gzip -6 is 117,337 (0.293) and gzip -9 is 115,419 (0.289).

So the narrow alphabet is a sharper discriminator, not the only one: a prose fixture catches the defect at 14%. What let it through was simpler — no arm bounded the ratio at all, on any corpus.

A test that asserts “it compressed” cannot find this. The arm that catches it (the_match_chain_reaches_past_the_most_recent_candidate) asserts a ratio BOUND on the colliding corpus, with the decode as its control — a smaller output is only good news if it is still the same bytes.

Separating the search from the code table

Comparing against gzip -6 cannot attribute a gap to the coder or to the matcher, because gzip changes the search and the code table together: lazy matching, a far longer chain, and dynamic blocks. zlib’s Z_FIXED strategy is the discriminator, because it forces fixed-Huffman blocks while the level still sets the search effort. On the same corpus:

zlib’s strongest matcher, restricted to fixed blocks, still only reaches 0.402 — so the code table does most of the work and no chain depth substitutes for it. rcp at 0.450 against even that meant the matcher was leaving about 11% on the table too: greedy, with no lazy matching. Both were real work.

Both are done. Dynamic blocks took the same corpus from 0.450 to 0.301 — past Z_FIXED level 9’s 0.402, as the table predicted, which is why they were built before the matcher was touched. Lazy matching then took it to 0.3003.

That second number is nearly flat because the 400 KB fixture is pathological by construction and is not what lazy matching is for. On a corpus of this crate’s own source and prose it is worth 2.7%, which is most of the remaining gap:

What is left is an optimal parse (zopfli, libdeflate), which rcp does not attempt.

0.402 is a measurement, not a floor. It is one strong matcher’s result, and an optimal parser emitting only fixed blocks could beat it; calling it “the floor for any fixed-Huffman encoder” would draw a mechanism from a single data point. MAX_CHAIN stays at 32 (the sweep’s knee) because chain DEPTH is what is nearly exhausted.

A bar calibrated downstream decays silently

the_match_chain_reaches_past_the_most_recent_candidate bounded the narrow corpus at < 0.53, chosen so the replanted chain defect (0.537) fails it. With dynamic blocks the defect compresses to 0.3338 — comfortably under the bar. The arm still passed, still named the chain, and could no longer detect the thing it was written for.

Nothing failed to announce this; the arm simply kept being green. It shows up only when you ask what the defect now measures, rather than what the fix measures, which is the question a bar tightens against. The arm therefore does not measure the matcher through the coder: it asserts on the LZ77 symbol stream — how much of the input matches cover — which no code table can flatter, and keeps a ratio bound beside it as the end-to-end guard.

The general form: a bar calibrated against one configuration decays when anything downstream of the thing it measures improves, and it decays silently, because improvement moves the number the safe way. Any arm asserting on an end-to-end number is measuring every stage between its subject and its assertion.

Incompressible input is not inflated

With only fixed-Huffman blocks, a literal costs 8 bits under byte value 144 and 9 above, so uniformly random input averages 8.4375 bits per byte — +5.44% measured on 200 KB of noise. rcp copy /tree tar://out.tar.gz over an OCI layout or a photo library is exactly that input, so a fixed-only encoder makes the .tar.gz larger than the .tar, under a name that promises the opposite, while every arm stays green: the stream is valid, gzip -d reads it, tar xzf unpacks it. Nothing there tests the direction of the size change.

Each block goes out stored, fixed or dynamic, whichever costs fewer bits — an exact count of each candidate including its own header, so the choice cannot be wrong about what it is choosing. Stored is what gzip has always fallen back to and costs five bytes per 64 KiB. The arm (incompressible_input_is_not_inflated) bounds the overhead at +1% and carries two controls, because “smaller” is not “correct”: the noise must decode back byte for byte, and text must still compress by 50x, or the encoder has quietly become cat.

4 MiB members

The stream is a run of independent 4 MiB gzip members rather than one member, which is what pigz -i writes and what RFC 1952 §2.2 allows. deflate() takes a whole slice, so one member would mean holding the archive in memory. It is also the shape the block pipeline, -rsyncable and parallel decode need: each starts from a format that already has independent blocks in it.

Compression on the ssh:// stream

-compress compresses each payload on the ssh:// stream with a raw-or-compressed flag per payload. LZ4 is the default; -compress.codec deflate and -compress.codec zstd are the two slow codecs. Measured through the counting stand-in that sits in ssh’s position, on 4.5 MB of log-shaped text (timestamps, hostnames, varying numbers – not one repeated line, which LZ4 handles unrepresentatively well):

The two slow codecs are within half a percent, and zstd’s ratio comes from its search, not its container. As first wired, the zstd encoder sent 960,872 bytes – 59% behind this crate’s own DEFLATE. Each change below closed part of that gap without changing the format:

So what zstd buys is interchange: the frames are ordinary zstd frames, so a peer, a pipe or an archive that already speaks zstd needs nothing from us. It is not a ratio sacrifice to use it.

The last digits of every figure here move by a few bytes between runs, because the count includes request paths and the scratch directory’s name changes. The comparisons are stable; do not read the units digit as a constant.

The negotiation is an ANSWER, not a request, and that asymmetry is the design. The client names a codec; the serve replies with the codec it will actually use. A serve that does not know the name answers LZ4 and the client follows its answer rather than its own request – because two ends disagreeing about how to read a payload is a corruption, while falling back is only a worse ratio. The caller asked for a smaller stream, not a different outcome. -compress.codec brotli is refused rather than quietly falling back.

Every codec field in HELLO is read as required. Within a protocol version every peer sends them (see the level section below for the full argument). The reply shows why a tolerant read is dangerous as well as useless: FOUR bools follow the codec there, so a tolerant read of an absent field would consume them instead – the same shape as a defect in hello where a short-circuited flag left every later answer one field late and sync silently recopied nothing.

Wire tags. zstd is payload tag 2 and DEFLATE is tag 3, and both non-LZ4 codecs ask with HELLO version 6, so a raw or LZ4 peer’s handshake is unchanged. zstd holds tag 2 because an earlier release used it; changing it would break interop.

Each codec carries its own expansion bound, used to refuse a dishonest raw_len before anything is allocated: 255 bytes per input byte for LZ4 (a sequence’s length bytes extend by 255 apiece), 1032 for DEFLATE (a length code reaches 258 bytes and the cheapest match is two one-bit codes under a dynamic tree – zlib’s own worst-case figure). The multiplication saturates: MAX_FRAME (16 MiB) times 1032 is about 17 GiB, fine in a 64-bit usize, a panic in a 32-bit debug build and a wrap in a 32-bit release one. The wrap is not a hole – raw_len is capped at MAX_FRAME first, so even the wrapped bound admits every legal claim – but it is a crash on a 32-bit target. The CRC-32 over the ORIGINAL bytes covers every codec, because a flipped bit in a compressed body usually relocates a match rather than failing the decode, so the decoder emits plausible bytes that are not the ones sent.

The LZ4 prefix probe’s timing arm compares like with like. Before a block is compressed whole, a 64 KiB prefix is probed; after eight unprofitable blocks the next sixty-four go raw unprobed. A rejected block is still copied whole into the encoder, so the arm prices that copy separately rather than charging it to the probe with nothing on the other side of the ledger, and takes the minimum of seven samples. Its control lands at 1.658 ms against a 1.659 ms whole-block pass.

zstd codec

-compress.codec zstd is an encoder that emits Huffman-coded literals and FSE-coded sequences, and a decoder that reads the format back, in the crate’s own code. That is FSE entropy coding, Huffman literals and a sequence format in both directions, in a crate whose only dependencies are cryptography; a thin encoder wearing the name would be a label rather than a capability, which is why DEFLATE came first.

The system library is an optional backend, not an alternative. --features system-zstd swaps the system libzstd in behind its stable one-shot C API. The crate’s codec is in every build. Frames cross between the two in both directions, and a unit arm asserts it on text, a run and noise.

The same codec writes .tar.zst. A .tar.zst is written as one frame per 4 MiB, through the pool and the content-defined cuts the .tar.gz writer already has, and read by sniffing the frame magic or a leading skippable frame (pzstd’s layout). It always uses the crate’s codec, not the system-zstd backend, so an archive is the same bytes from every build. The ratio is gzip’s rather than zstd’s: 991,256 bytes against zstd -3’s 922,291 on 3.3 MB of this crate’s text. Closing that gap means a better match search, not a different container.

Interop is checked against the reference, not against itself

Both directions are checked against /usr/bin/zstd rather than against each other – the reference decodes what this encoder writes, and this decoder reads 15 reference frames (five corpora at three levels) byte-for-byte, plus 280 more in a boundary sweep over the lengths where the format changes shape, at two levels crossed with and without a frame checksum.

Five ordinary corpora at three nearby levels is ONE compressor on typical files, and several paths this decoder parses do not appear in it. The boundary sweep, the mixed-block-type frame, the corrupted-input fuzz arm and the verified content checksum exist for that reason. The claim is “agrees with the reference wherever we have looked, and we have looked at the edges”, which is weaker than “reads the whole format” and is what the evidence supports.

The fuzz arm is differential. Counting 0 panics over thousands of hostile inputs says nothing about the ones that DECODE – a confidently wrong answer passes as a success. Every accepted input is also put to the reference, and over 4,710 corrupted, truncated and random inputs the split is: 129 accepted and byte-identical to the reference, 0 accepted only by us, 4 refused only by us. Those four are one cause, and the reference is the permissive one: RFC 8878 says the low two bits of Symbol_Compression_Modes are reserved and must be zero, and libzstd masks them off. Refusals the reference disagrees with fail the arm unless they name a rule like that one, so “we are stricter” cannot be used as a blanket excuse for a real interop bug.

Two things that arm measured on the way:

• The content checksum silently empties a naive fuzz corpus. Once the encoder writes a checksum, every corrupted frame fails verification instead of decoding, and the accept count over the original inputs went from 129 to zero. That is the checksum working, and it also blinds the differential, which can only compare what somebody accepts – so the corpus mutates unchecksummed frames too, which is what a peer running zstd --no-check would send anyway.
• A length check would not catch the replant. The control for this arm is a one-byte offset error in the match copy; it produces 3,000 bytes where the reference produces 3,000 different bytes. Same length, different content, no error raised by either decoder on its own.

Window bounds

The window bound is enforced at every append inside the block decoder, with a post-block check kept as a backstop that should never fire. Checking only after a block has expanded gives the right refusal after the memory has been spent.

Each window refusal is pinned to its own wording. Both refusals say window allows, so an arm asserting only that substring keeps passing when the bound moves and the arm starts exercising the other check. The offset case has its own corpus (a unique prefix, 140 KB of filler, the prefix again: a match reaching back ~145,000 bytes, refused at a 128 KiB window and accepted at 256 KiB).

Declared-window cap. zstd --ultra -22 --long=31 writes a frame whose declared window the reference refuses: zstd -d answers Window size larger than maximum : 2147483648 > 134217728. The cap is > 1<<27, it applies to single-segment frames too (where the window IS the declared content size), and both sides of the boundary are asserted – a hand-built frame declaring exactly 1<<27 decodes in both, 200,000,000 is refused by both. Building those by hand is cheap: RLE blocks cost four bytes per 128 KiB, so a 200 MB declaration is a 6 KB frame.

Conformance checks run in both directions. “What do we refuse that the reference reads” (the differential’s stricter pile, the DELIBERATELY_STRICTER list, the conformance sweep) is half a suite. “What does the reference refuse that we read” is the other half, and it is the question that found the window cap. Which half is missing is invisible from inside a suite written in one direction.

Coverage of the decoder’s branches

The hostile corpus is required to reach eight named conformance refusals rather than merely to produce errors. It reaches 24 distinct ones; the rarest of the named eight fires 6 times, which includes the leftover-bits check.

Three decoder paths the encoder cannot write – treeless literals, single-stream Huffman and repeat-mode sequence tables – are exercised by two corpus families: one large skewed input, so the reference spans several blocks per frame and starts reusing trees and tables, and two small ones, whose literals sections fit the single-stream form. Without them those three branches were reached zero times across 268 sweep frames (RLE sequence tables four times); they parse attacker-supplied bytes, and agreement over the other frames said nothing about them. The sweep fails by name if any branch stops being reached. The grid is 70 corpora / 280 frames, and the interleaved FSE decoder covers 68 entries.

The block type and the literals-section type are counted too:

raw literals 140 | RLE literals 2 | RLE block 4 | raw block 102

All four are covered, and the counters stay asserted so a corpus that stops producing one of these shapes says so rather than quietly narrowing. A replant that removes one branch() call fails the sweep, so the assertions are live rather than decorative.

Frame-level rules

Every frame is decoded. Concatenating frames is legal zstd, so ignoring bytes after the first frame silently drops a whole trailing frame: with two frames in, the reference returns 700 bytes and a first-frame-only decoder returns 320, with no error – a correct-looking decode of the wrong bytes, in the function every caller uses. Bytes that are not a frame are refused rather than shrugged at, and skippable frames are skipped.

A frame naming a dictionary is refused, at all three ID widths. Measuring Dictionary_ID and stepping over it decodes the blocks after it and matches the checksum – 1,024 bytes returned where zstd -d answers Decoding error (36) : Dictionary mismatch. The corpus has a family for this: the existing bit flips do set the dictionary flag, but setting it without inserting the ID shifts every later field, so both decoders refuse for reasons unrelated to dictionaries and the input lands in the refusal pile looking like coverage. Replanting the defect moves we_are_laxer from 0 to 3.

A skippable frame’s Frame_Size counts User_Data alone. RFC 8878 3.1.2 defines it that way, so the advance is 8 + n. Reading it as including the 8-byte header would be an off-by-eight that desynchronizes everything after the first skippable frame; the reference settles it: with the field at 11 and nineteen bytes on the wire, zstd -d lands on the next data frame in all five arrangements, where the other reading would land eight bytes early in every one. The five arrangements are in the arm, and the RFC sentence is quoted where the advance is computed.

Single-segment frames reach the decoder through a file-based path. A piped zstd cannot know the input length, so it clears Single_Segment_flag every time; handed a FILE it sets the flag at all three content-size widths – 10 bytes gives descriptor 0x24, 1,000 gives 0x64, 300,000 gives 0xa4. It is tempting to conclude from piped measurements that the reference never emits single-segment frames; it does. Because reference() in the test module pipes, every one of the ~400 reference frames in the other arms is descriptor 0x04 or 0x00, and no amount of adding corpora changes that: the shape is chosen by how the reference is invoked, not by what it is fed. A file-based path and an arm requiring all three widths cover it; 18 frames decode byte-for-byte.

Decoder defects found by reference interop

Four of them, and each was invisible to the others – which is the argument for the five-corpus interop arm rather than one sample:

The through-line is that three of the four produced plausible output or a green suite rather than an error. A round trip between an encoder and a decoder that share a table cannot see a wrong table. Only the reference can.

DEFLATE compression level

-compress.level N, 1..9 in zlib’s sense, default 6. It moves two numbers inside the matcher – how far down the hash chain a search walks, and how long a match has to be before the lazy arm stops looking for a better one at the next byte – which is the same pair zlib spells max_chain_length and max_lazy_match.

Level 6 is asserted to be byte-for-byte the encoder without a level knob. A default that quietly changes output is a regression for every caller who never asked for one, so deflate_at(data, Level::of(6)) == deflate(data) is the first assertion in the unit arm rather than a comment.

Measured on the 400 KB colliding corpus the chain arms already use:

Level 9 is below gzip -6’s 117,337 on this corpus and within 1.1% of gzip -9’s 115,419 – so the residual gap to zlib at the top of the range is about a percent.

Levels 3 and 4 are not strictly ordered (122,739 against 122,785): a deeper chain can pick a different match that codes a hair worse. The arm therefore bounds 1 against 9 end to end rather than asserting monotonicity it does not have.

The level never crosses the wire as a stream property, and that is the whole difference between it and the codec. A DEFLATE stream is self-describing – the decoder reads the block headers and the code tables out of the stream itself – so the two ends do not have to agree on a level the way they must agree on a codec. It rides in HELLO only as a REQUEST, so a number typed on one command line reaches the other side’s encoder.

The trailing HELLO fields are required, not tolerated. VERSION is checked for equality, so a peer predating codec or level is already refused by version; within a version every peer sends both. The only way a tolerant unwrap_or read could fire is a TRUNCATED frame, and a truncated frame must not read as an absent field. Such a tolerance would support no honest peer and would silently turn a corrupt HELLO into a plausible session.

A bad value is refused at the terminal and defaulted from the wire, on purpose. The command line REFUSES 0, 10 or six. The serve answers an out-of-range level with the DEFAULT. A typo at a terminal is a question the caller can still answer; a field in a frame from a peer is not, and working at the default is a better answer there than dropping a connection over a number that cannot affect whether the result decodes. Clamping to 1..9 was rejected: clamp(1, 9) sends 0 to 1 and sends 10, 255 and 1000 to 9, answering the one input a caller cannot have meant with the most expensive setting on offer. The safe reading of a value a peer should never have sent is the one this side would have used had the peer said nothing.

The contract arm pulls rather than pushes. A push counts bytes this process compressed, so a level that never left the machine would still pass; pulling counts bytes rcp serve compressed. Measured: level 1 793,848 bytes out of the serve, level 9 599,604. Two different totals are consistent both with the level having crossed HELLO and with the serve having been configured that way itself, and the numbers alone cannot separate those. So the serve logs the level it actually used and the arm reads it back; the stand-in starts rcp serve with no arguments, so the only place that number can have come from is the frame. A serve patched to ignore the field sends the same number twice and fails the arm.

LZ4 has no level – it is the single-pass block format – so -compress.level without -compress.codec deflate is refused. Accepting and ignoring it was considered and rejected. The argument for it is that refusing makes the level a trap for anyone who has not also typed the codec; but the trap is the SUCCESS: exit 0, no warning, no reply field echoing the level actually used, and a caller who has every reason to think the number did something. It would also contradict the rule beside it – -compress.level 10 is refused because a typo at a terminal is a question the caller can still answer, and the person who omitted the codec is at that same terminal. Two rules in one option cannot point opposite ways.

Skipped arms

The suite prints a line whenever an arm cannot run – 37 such sites – but cargo test captures stdout for passing tests, so a skip that fires is invisible in a normal green run. Measured with --nocapture across the lib and integration suites: one, SKIP the unreadable arm: running as root, which reads a 0o000 file, which is a property of the container and not of the code. Everything else – every zstd and gzip reference half, every python3 stand-in – ran. A default run cannot report this count; --nocapture can. A skip is a check that did not happen, never a pass.

Sparse tar members

Without sparse support, tar:// writes a hole as the zeros it stands for. Measured on a 64 MiB file holding 128 real bytes and occupying 8 KiB on disk:

Read the size column, not the time one. On this machine the fixture lives on tmpfs, where writing 64 MiB of zeros is nearly free: the release-build dense pack of the same apparent bytes takes 0.42 s against the sparse path’s 0.30 s, which is not a headline. The 67 MB that no longer exists is the result – on a real disk, in an artifact store, or across any of the network transports this crate has, that is the whole cost. The one time figure worth quoting is verify through the archive: 0.05 s against a dense member, 0.02 s against the sparse one, because the reader skips holes instead of comparing zeros to zeros.

The format is read off the reference

pax reserves the GNU. keyword namespace and says nothing about what goes in it, so the specification here IS GNU tar. What 1.35 writes, verified by od:

x header at ./PaxHeaders/<base>     GNU.sparse.major=1  .minor=0
                                    .name=<real name>   .realsize=<apparent size>
0 header at ./GNUSparseFile.N/<base>   size = map text + extents, block-padded
data:  "<count>\n" then "<off>\n<len>\n" per extent, NUL-padded to 512, then the extents

Two details that are not obvious and are copied rather than reasoned about: the member name is a decoy, so a reader that does not know the extension extracts something visibly wrong rather than a silently truncated file; and GNU closes the map with a zero-length extent at the file’s end. The reader here does not need that terminator – realsize already says where the file stops – but writing what the reference writes is the cheap side of an interop question nobody wants to re-answer.

One extended header per member, and why rcp’s own tests could not see it

The member header must not go through write_header. That function emits its own pax header whenever a field will not fit ustar – a sub-second mtime, or a decoy name past 100 bytes – so a sparse member written through it carries two x headers in a row.

That is not two sets of records. GNU tar resets its pending records at each extended header, so the second one erases GNU.sparse.*, and tar xf produces:

GNUSparseFile.2452/hole.bin   4672 bytes

The decoy, taken literally, with no error reported. rcp’s own reader accumulates pax records across headers, so it merges the two and is perfectly happy: every rcp-side assertion – round trip, digest, verify, hole preservation – passes. Only the reference catches it, which is the argument for running the other implementation rather than trusting a green suite of one’s own. The member header is therefore written inline with no pax fallback, and every record the member needs is folded into the single sparse header.

The replant is faithful and fires exactly there: restoring the write_header call leaves every rcp-side assertion green and fails at the tar-interop half with

tar (GNU tar) 1.35 did not produce hole.bin; it extracted:
["GNUSparseFile.3942", "GNUSparseFile.3942/hole.bin"]

The 8 MiB decision window

A member’s header must be written before its bytes, and a sparse header needs the whole map, which is not known until the last byte. So the choice cannot be made up front, and deferring it to the end would mean buffering every dense file in the tree. The sink holds the first 8 MiB while it waits to see a hole:

• a hole in that window -> sparse, and the non-hole bytes spill to a temp file (bounded by the stored size, which is exactly what the archive will hold, not by the apparent size)
• 8 MiB of solid bytes with no hole -> the ordinary layout, header written then, bytes streaming straight through as before
• a hole that arrives after that -> written as the zeros it stands for, correct and larger

A file that is sparse is sparse by allocation pattern, not by coincidence at the tail, so the early window catches the real cases. The third bullet is asserted by an arm rather than left as a hope: a 12 MiB file solid for its first 9 MiB packs to more than 11 MiB, and still round-trips to the same digest.

The arms, and what each control is for

Three refusable shapes in the map, four in the validator, and two controls that are the point rather than decoration:

• a tree with no holes must not acquire a sparse header and must not shrink. Without it, a writer that marked every member sparse – or one that simply lost data and wrote a short archive – passes every assertion about the sparse fixture.
• the map the writer emits must validate. Four refusal assertions pass trivially against a validate that refuses everything, which would make the reader useless while the arm stayed green.

Replanted three ways, each firing on its own line: the old flat-hole path fails the archive size; the two-pax-header defect fails the tar-interop half; a reader that ignores GNU.sparse.* fails the round trip.

A green suite can hide a test that stopped running. Inserting the map unit test above an existing one left the existing test’s #[test] attribute attached to the new function and stripped it from the old: two registrations of the new arm, zero of a_member_that_climbs_out_of_the_archive_is_refused. The suite stayed green. It was found by reading the test-name list rather than the pass count: confirm the count is plausible before concluding anything from a green run. 407 tests, up from 405 by exactly the two arms added.

Reading all four GNU sparse encodings

A reader that handles only pax 1.0 and relies on a version guard to refuse the rest does not work, because the older pax forms carry no version at all. Measured, against archives GNU tar 1.35 wrote from the same fixture, with only a 1.0 reader in place:

Silent, successful, and a different file. The mechanism is one key: GNU.sparse.major arrived with 1.0, so 0.0 and 0.1 carry no major at all, and the guard that refused “a sparse format this reader does not have” never fired for them. They fell through to the ordinary path, which read the extents as the whole file.

All four are read, which is the only sound resolution – “refuse what we do not know” does not work when the thing you do not know looks exactly like something ordinary.

0.0 forced a second pax reader: records have to be kept in order and with repeats, since a hundred extents collapsed into a BTreeMap become one – the last – and the file that comes out is again a different file with nothing reporting it. pax_list is that reader and pax is built from it.

The arm is a sweep over all four, and it asserts the COUNT (read_back == 4) rather than > 0, because a sweep that quietly shrank to the one case that works is the failure this exists to catch. Three replants, each firing where it should: 0.0/0.1 falling through fails with pax 0.0 (repeated offset/numbytes records): wrong length; S not being a file kind fails the copy; collapsing the pax records into a map fails the copy.

Only 1.0 is WRITTEN, deliberately – it is the portable form, GNU tar and bsdtar both read it, and writing four encodings of one idea would be four times the surface for no reader that gains anything.

Edge cases

All four files pack to 18,432 bytes together; gzipped, 523 bytes. Sparse composes with .tar.gz and with parallel decode, and tar -xzf reads the result with the hole intact. None of these cases failed. The all-hole case was the one expected to break – its extent map is nothing but the terminator – and it does not.

Parallel gzip decode

A gzip member does not say how long it is. Its compressed length is known only once it has been inflated, so the boundaries of a multi-member stream look irreducibly serial: to find member 2 you must finish member 1.

What breaks the chain is that member starts are findable, if not decidable. Measured on an 18.6 MB archive holding 83 MB of log text, written by this crate:

The cheap filter removes 94% of the noise and leaves one false survivor. That ratio is the whole feasibility argument: speculate on every survivor, and the waste is one member in twenty-one.

The chain, not the speculation, decides the output. Results are collected by offset; then the member list is walked exactly as the serial gunzip walks it — start at 0, take that member’s consumed length, look for a result at the offset it lands on, inflate on demand if there is none — and only members ON that walk are concatenated. A false candidate is simply an offset the walk never visits. Speculation can waste work; it cannot change the answer.

Measured, release, four cores:

End to end through rcp copy tar://a.tar.gz DIR the unpack goes ~1.93 s to ~0.87 s, but that figure is not clean attribution — -parallel 4 also parallelizes the file writing. The isolated decode numbers above are the ones that mean something.

A counter proves the parallel path runs

gunzip_with falls back to the serial path on anything it does not like, so an arm asserting “the output is identical at 8 workers” passes while the parallel path is never entered. A #[cfg(test)] counter reports (candidates inflated, of those used by the chain), and the arm asserts both.

The counter caught a defect on its first run. The gate was data.len() < 2 * GZIP_BLOCK — and data is the compressed stream, so five 4 MiB members of ordinary text, which compress to well under 8 MiB together, fell under the threshold and went serial. Every equality assertion passed at every worker count, because the serial path is what they compared against. The gate is the candidate count.

Adversarial arms, and which property each one proves

The obvious adversarial arm plants plausible gzip headers inside a member’s compressed bytes — magic, method 8, no reserved flag bits — and checks the output is unharmed. The fixture needs care: surrounding each decoy with a run of 0xAA makes the region compressible, so deflate emits matches and the bytes never reach the stream. Measured: 200 occurrences of 1f 8b, 4 surviving the filter, and those 4 were the real members. Pure random data stores rather than compresses, and then the fixture works: 222 occurrences, 21 surviving, 3 real members.

That arm proves less than it looks like. Replanted two ways — a chain that trusts every candidate, and no header filter at all — it stays green both times. Its decoys are random bytes behind a valid-looking header, so gunzip_member rejects them at the CRC. They never become results, and reassembling the successful results in offset order happens to be right. The header filter and the CRC do the work; the chain is never tested.

So a second arm plants a complete, valid gzip member inside another member’s stored block: correct magic, correct deflate stream, correct CRC-32, correct ISIZE. Every check short of the chain accepts it. The chain is the only thing keeping DECOY CONTENT out of the output, and the replant that makes the chain trust its candidates fails by name.

The weaker arm is kept, with its doc comment saying exactly what it does and does not prove, because “these decoys are rejected by the CRC” is a real property worth holding — it is not the one the arm’s name suggests.

To keep a failure readable, the arm compares the two 12 MB vectors by length and content and reports the first differing index; assert_eq! on them prints 55 MB on failure.

Memory

Peak memory is the sum of every candidate that DECODED, not just the ones the chain keeps: a false candidate’s output sits in the map until the call returns. The expectation was roughly 2x. Measured with VmHWM on the 83 MB archive:

Unchanged — and the reason is the remove that was added for SPEED. The chain walk takes ownership of each member as it goes, so the map shrinks exactly as the output grows and the peak is about one copy rather than two. Reverting that optimization would cost memory as well as time.

The hostile case, and why it is hard rather than impossible

A hidden member amplifies memory only if it both SURVIVES being embedded and EXPANDS. Those requirements fight, and the measurement says so:

To survive, a decoy’s bytes must reach the stream literally, which needs its containing block to be stored — and anything compressible enough to expand usefully tips that block toward being compressed instead. Stored blocks also carry a header every 64 KiB, which cuts any longer decoy in two. The decoys that DO survive elsewhere in these arms are tens of bytes, precisely because they are too small to tip anything.

“Hard for an adversary” is not a bound, so there is one: speculation stops once the decoded total passes limit, the ceiling the function was already given. Correctness does not depend on where the cap falls, because the chain inflates on demand whatever the pool did not reach — and that is asserted rather than assumed, by an arm requiring the limit to mean the same thing on both paths, met and exceeded. Replanted by making the chain give up instead of inflating on demand: it fails with “the parallel path accepted a limit the serial path refused”, which is the silent truncation the arm exists to catch.

Building

See platform validation for Linux ARM64 work and the Windows portability blockers. A successful cross-check is not native runtime validation.

cargo build --release                    # the default build; see below for --offline
RUSTFLAGS='-C target-feature=+crt-static' cargo build --release   # static binary
cargo test
cargo build --release --features rustls  # https:// S3 endpoints; adds rustls and ring

All commands run from the repository root. The default build compiles this crate, the audited open-source cryptography crates it uses – RustCrypto sha2, hmac, md4, md-5, aes and cmac, and the official blake3 – and the small support crates those need (digest, cipher, hybrid-array, typenum, cpufeatures and the like). CI holds the default build to exactly that list with cargo tree, so a new dependency is a reviewed change to the allow-list rather than an accident. blake3 compiles its assembly and C SIMD kernels with the cc crate at build time, so a C compiler must be on the host, as it already is for the rustls build’s ring.

An air-gapped host needs those crates before --offline works, and cargo resolves optional dependencies too, so the lockfile’s TLS entries need the registry index even when the rustls feature is off. Either copy the registry from a connected host (run cargo fetch --locked there and carry ~/.cargo/registry across), or use a cargo vendor directory. Then cargo build --release --offline --locked works as above. The rustls build adds rustls and ring, the one thing the standard library does not have, and is the one to ship where buckets live behind HTTPS.

Hardware acceleration

sha2, aes and blake3 pick their CPU paths at run time: SHA-NI and AES-NI on x86-64, the ARMv8 SHA-2 and AES instructions on ARM64, and SSE4.1, AVX2 or AVX-512 (NEON on ARM64) for BLAKE3, with portable code where the CPU has none of them. rcp version reports what SHA-256 found. RCP_NO_SHA_NI and RCP_NO_AES_NI, which used to force rcp’s own portable code, are still accepted and have no effect: the crates’ selection cannot be overridden at run time.

Choosing endpoints

Each endpoint is a Cargo feature, and all of them are on by default: nfs, s3, smb, sftp, remote (ssh://, rcp:// and rcp serve), dav, hdfs, https, oci and tar. None adds a dependency, so turning one off only makes the binary smaller and its attack surface narrower. A build names the ones it keeps:

cargo build --release --no-default-features --features nfs,s3

Every scheme still parses in such a build, so a URL for an endpoint that was left out is refused by name rather than read as a local path:

$ rcp copy /data smb://nas/share/data
ERROR: smb://nas/share/data: this rcp was built without the `smb` feature; rebuild with `--features smb`

rcp serve needs remote and rcp show HOST needs nfs. Options that only configure a left-out endpoint, such as -smb.sign without smb, are accepted and have nothing to configure. The contract suite in tests/contract.rs covers the default build; CI’s features job checks that the library and binary build with each feature alone, with none, and with every pair.

The release build targets x86-64 GNU/Linux and dynamically links the host’s GNU libc. Its archive includes RUNTIME.txt, which records the build environment and required glibc symbol versions, and THIRD_PARTY_LICENSES.txt for the Rust dependencies. Check those requirements on the destination host; compatibility with older distributions is not implied by the target name. Kerberos needs the system GSSAPI library and a valid ticket. Safe job cancellation uses pidfds on Linux x86-64 and AArch64 and requires kernel support for pidfd_open (Linux 5.3 or later); an unavailable or blocked syscall is reported as a refusal.

The +crt-static commands above and below are custom builds for local interoperability testing, not the release configuration. They need the target’s static C runtime at build time and cannot load the system GSSAPI library for Kerberos. Distributing a custom static GNU build also requires handling its C runtime’s license and source obligations; the Rust dependency notice bundle does not cover that runtime.

Against real servers

The contract suite talks to in-process stand-ins. scripts/interop.sh puts each network endpoint in front of the server people actually run, one container each: VersityGW (S3), the CNCF distribution registry (OCI), Apache httpd with mod_autoindex (HTTPS sources) and mod_dav (WebDAV), OpenSSH (SFTP, and ssh:// with this build’s rcp serve on the far side), Samba (SMB, plain and signed), NFS-Ganesha (NFSv3 and v4.0) and Apache Hadoop (WebHDFS). For each it copies a fixture up, verifies it, copies it back, and diffs both the round trip and the server’s own copy against the fixture, so a defect that is symmetric in rcp still shows up. The fixture is small and hostile: an empty file, sizes one byte either side of the read and write boundaries, a 17 MB file, a 200-byte name, spaces, %, #, + and non-ASCII names, and an empty directory. Needs Docker, and a static build for ssh://:

RUSTFLAGS='-C target-feature=+crt-static' \
  cargo build --release --features rustls --target x86_64-unknown-linux-gnu
RCP=target/x86_64-unknown-linux-gnu/release/rcp scripts/interop.sh          # all of them
RCP=target/x86_64-unknown-linux-gnu/release/rcp scripts/interop.sh smb nfs  # some

KEEP=1 leaves the containers and the work directory for a look afterwards. The Interop workflow runs one job per endpoint on every pull request and weekly.

Across hosts, on k3s

-nodes, -via and rcps:// need more than one host, so interop/k3s/ stands them up as pods on a k3s cluster with an Ansible playbook, then interop/k3s/tests.sh runs four scenarios from client pods:

• fleet: an S3-to-S3 copy spread with -nodes over three worker pods. Each worker must have run at least one batch, and the copy must verify and read back identical.
• relay: a copy, verify and read-back to ssh://hop-b with -via ssh://hop-a. A NetworkPolicy admits only hop pods to hop-b, and the client’s own dial to hop-b is the control: it must fail, so a pass went through the hop.
• mtls: the same round trip over rcps:// to a serve that requires a client certificate from the suite’s CA. No certificate, a certificate from another CA, and a serve the client does not trust must each be refused, and the serve must still answer afterwards.
• krb: NFS-Ganesha exporting with RPCSEC_GSS only, against an MIT KDC in the same pod. A copy, verify and read-back under krb5, krb5i and krb5p, and krb5p over NFSv4.0, each compared against the server’s disk. AUTH_SYS and a client without a ticket are the controls, and must be refused. The client runs a dynamically linked rcp, the only kind that can load the system GSSAPI library.

RUSTFLAGS='-C target-feature=+crt-static' \
  cargo build --release --features rustls --target x86_64-unknown-linux-gnu
cargo build --release --features rustls     # the dynamic build the Kerberos client runs
ansible-playbook interop/k3s/playbook.yml   # uses the cluster on $KUBECONFIG, else starts k3s
interop/k3s/tests.sh                        # or: tests.sh relay krb
ansible-playbook interop/k3s/playbook.yml -e state=absent   # delete the namespace

A cluster already answering on the kubeconfig is used as it is. Otherwise the playbook runs $K3S_BOOTSTRAP if set, or downloads the pinned k3s release and checks it against the release’s own checksums. The fleet, relay and mTLS pods run the static binary from a hostPath mount; the Kerberos client uses the separate dynamically linked build so it can load GSSAPI. A rebuild needs a re-run of the playbook. The Interop workflow runs this as its k3s job.

Platform validation

This is a development support matrix, not a list of published binaries. Releases publish the Linux x86-64 GNU build only; Linux ARM64 and Windows x86-64 are built from source. Candidates are development builds, not release assets.

Architecture

Keep copy, verify, scan and sync against the existing Tree and Sink ports. Platform filesystem, process and loader code belongs in driven adapters. platform centralizes Linux open-flag ABI constants used by the local tree, catalog, CLI log and io_uring adapters. AArch64 uses O_NOFOLLOW = 1 << 15 and O_DIRECTORY = 1 << 14, whereas x86-64 uses bits 17 and 16. These differences are specified by the Linux ARM64 UAPI header. The regression test checks kernel behavior, including that a rejected file-link open leaves its target bytes intact.

Sink::raw_fd now returns a platform kernel-copy capability: a Unix descriptor on Unix, and an uninhabited type on Windows. Windows sinks return None, selecting the generic thread engine without interpreting a Windows handle as an integer fd. The new Tree::validate_paths hook refuses case-only source/destination aliases before planning can treat one Windows entry as both a copy and a removal.

Native ARM64 qualification

A native run passed on GitHub’s standard ubuntu-24.04-arm runner. The run and the commits it tested predate this repository’s current history, so they are not linked here; rerun portability.yml to reproduce it. The script asserted uname -m = aarch64 and native Rust host aarch64-unknown-linux-gnu (Rust 1.87.0). This is native execution, not emulation.

Formatting and all-target/all-feature Clippy passed. The bounded s3,smb,dav,https,rustls build passed 362 library tests and 2 CLI unit tests. Default-feature architecture/reference suites passed 5 hexagon, 7 description, 1 documentation and 3 xxHash tests. All 13 exactly selected release/TLS CLI contracts passed: copy/verify, every engine, sparse files, resume, sync, deletion, hard links, xattrs, version/crypto vectors, MCP and job cancellation. Protected file/directory symlink opens and SHA-256 backend/FIPS vectors passed natively. The ordinary scope does not execute NFS, Kerberos or OCI realm contracts and does not qualify all native protocol peers.

The initial Ubuntu release/TLS candidate is ELF64 AArch64. Its direct dependencies are libc.so.6, ld-linux-aarch64.so.1 and libgcc_s.so.1; GSSAPI stays dynamically loaded. Its required symbol versions include GLIBC_2.39. It is an Ubuntu 24.04 development candidate, not proof of UBI 9/10 or GLIBC 2.29 compatibility. The archive checksum at the stated commit is 60043241711fbcab4cb4e46890d4d9b450519d527dcd16969ce535ffa21f5e32.

Reproduce on a native AArch64 GNU/Linux host with Rust 1.87, procps, zstd and libzstd development headers using bash scripts/test-linux-arm64.sh, or dispatch portability.yml with scope=ordinary-arm64. Full ARM64 protocol/KDC qualification and Windows qualification have independent manual scopes. No candidate is uploaded or released by this recipe.

Older-GLIBC ARM64 packaging

A native ABI run (which, like the run above, predates the current history) built the unchanged application with official Rust 1.90.0/Bullseye on the existing native ARM runner. The builder uses GLIBC 2.31 and the candidate’s maximum required symbol version is GLIBC_2.29. The independent native runtime is UBI 9.8/GLIBC 2.34. Its exact-binary copy/verify, isync, MCP ping, missing-input failure, deletion preview and forced deletion smoke passed, and the host GSSAPI library loaded. This establishes ordinary runtime compatibility on UBI9; it does not claim functional runtime testing on GLIBC 2.29, all protocol/KDC peers, UBI10 or FIPS.

The ARM supplier manifests are pinned separately from the existing x86 manifests:

• Builder: docker.io/library/rust@sha256:c90f06dc32cb5ba7716c1e65fc2c63b4a430580bd8b6a676e1028aed0ed53e5d.
• Runtime: registry.access.redhat.com/ubi9/ubi-minimal@sha256:a28d61d9531d13cb1c0a70d2e9f214aee029db73b614149122747905d973de74.

The builder’s default Linux protocol features plus rustls are compiled; the shared build script keeps its x86 default and accepts aarch64-unknown-linux-gnu. The gate validates ELF64 machine and interpreter, rejects GLIBC_PRIVATE and requirements above 2.34, and retains the observed maximum. Six gate tests passed, including newer/private ABI and wrong architecture/interpreter refusals. Existing x86 Container CI also checks its real default-target binary.

Binary SHA256 for this run is 081c468e7dcd5b7dacaf20cba9bc1cad00677abaf1f49c5f16532cdc4df1dc4b. Reproduce with manual portability.yml scope arm64-abi. The evidence artifact retains the exact binary, Cargo lock/metadata and notices, supplier configuration, RPM inventory, GLIBC gate output, toolchain and command results. The final native package run passed the same build/runtime gates and produced archive SHA256 c0186acac7fb5459af7e1ae1cb95c35e32e1f2e1e619d0cc67c1cb7f9e754da8. Both independent builds produced the same binary SHA256 recorded above. The downloaded archive was checked for exactly six regular files, licenses, ELF64 AArch64 identity and matching archived, downloaded and native-smoked binary checksums. The candidate archive includes licenses, runtime requirements and checksums; it is not published as a release. The UBI package lists name packages without versions, so a superseded RPM no longer breaks the build; a new base image digest or package set still needs renewed native qualification.

Windows candidate scope

Build with Rust 1.87 and the existing MSVC/Windows SDK tools:

cargo build --locked --release --no-default-features --features s3,smb,dav,https,rustls
./scripts/test-windows.ps1 -BuildCandidate

The default all-protocol feature set remains unchanged for Linux. On Windows, NFS, HDFS, SFTP, remote rcp, tar and system-zstd produce an explicit compilation error instead of bypassing their platform/authentication requirements. OCI can be compiled separately; it is not included in the qualified Windows candidate recipe.

The NTFS adapter opens each component relative to a pinned directory handle with reparse processing disabled, refuses traversal/device/stream names and junctions, and enumerates by handle. Other filesystem identities and UNC local mounts are not qualified. Files are staged exclusively, flushed and atomically renamed. Foreground termination keeps the old destination and the catalog can resume. This is not a transaction over the whole tree. Abrupt termination can leave an abandoned stage file; ordinary aborted sinks remove their stage files. The attempted NT delete-on-close toggle returned ERROR_NOT_SUPPORTED on this host, so it is not part of the qualified adapter.

Sparse output and native hard links are supported. Times use NTFS 100 ns precision. The adapter does not translate POSIX owners, modes, ACLs or xattrs to Windows security descriptors: owner/xattr refusals and skipped verification checks remain visible, and chmod/chown are refused. Catalog/cache creation uses a protected owner-only DACL. Existing operator-selected ACLs are not rewritten.

Detached jobs, worker fleets, MCP, diagnostics and mount reports are explicitly unsupported on Windows. They need separate process-tree lifecycle and platform adapters; no numeric-PID killing, post-spawn cancellation race or no-op security check is substituted. -nocache is refused. Delta and io_uring select the existing whole-file/thread fallbacks, with capabilities and engine warnings visible.

SMB uses the Windows system cryptographic RNG; a failure refuses authentication. No clock-derived nonce fallback exists. Large Windows sends are bounded in chunks so the existing socket/absolute deadlines remain effective. Windows TLS uses an explicit PEM bundle through RCP_CA_BUNDLE, SSL_CERT_FILE or AWS_CA_BUNDLE; there is no Windows certificate-store integration or verification downgrade.

Original audit and remaining semantics

The compiler audit used the default endpoint features; no endpoint was disabled to make the result look successful. It found Unix-only imports in catalog, tree, ports, anytree, diag, jobs, MCP, io_uring and protocol stand-in servers. Linking additionally requires replacements for Linux filesystem syscalls, dlopen/dlsym, POSIX signals/session creation and xattrs.

The original default-feature audit had 158 errors. The implemented subset addresses filesystem primitives and local operations; the remaining full-port requirements are:

• Native qualifications for unsupported filesystem identities and UNC mounts, and explicit translations if POSIX or Windows ACL preservation is added.
• Pinned process identities and process-tree cancellation using Windows handles and job objects, retaining MCP output isolation and resumable catalogs. Do not substitute arbitrary numeric-PID termination for the Linux pidfd guarantee.
• Existing SHA-256 and AES x86 CPU dispatch already has software fallbacks for non-x86 targets; validate their vectors on native ARM64.
• An explicit Windows GSSAPI library/provider contract or a clear Unsupported refusal for Kerberos. Linux continues loading host GSSAPI dynamically. Windows SSPI is not ABI-compatible with that loader. Validate native zstd linking when the optional system-zstd feature is enabled.

Native Windows x86-64 qualification used Rust 1.87.0, MSVC 14.44 and Windows SDK 10.0.26100.0 on NTFS. Formatting and Clippy with warnings denied passed. The optimized candidate subset passed 328 library tests, 2 CLI unit tests, 5 architecture tests and 10 Windows functional tests, including foreground termination with catalog resume. Enabling rustls passed all 11 Windows functional tests, including successful explicit-CA transfer and rejection of a missing CA bundle. The recipe uses the same feature selection in CI and creates candidates only after the selected tests pass.

Reproducing checks

cargo fmt --all --check
cargo clippy --locked --all-targets --all-features
cargo test --locked --test hexagon
cargo test --locked
rustup target add aarch64-unknown-linux-gnu
cargo check --locked --target aarch64-unknown-linux-gnu --all-targets
# On an x86-64 Debian host with gcc-aarch64-linux-gnu and libc6-dev-arm64-cross:
CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc \
  cargo build --locked --target aarch64-unknown-linux-gnu
aarch64-linux-gnu-readelf -h target/aarch64-unknown-linux-gnu/debug/rcp
aarch64-linux-gnu-readelf -d target/aarch64-unknown-linux-gnu/debug/rcp

The all-target cross-check proves type checking. The separate cross-build completed with Rust 1.98.1 and Debian 12’s GCC cross-toolchain; ELF inspection reported ELF64 AArch64 and direct dependencies on libgcc_s.so.1 and libc.so.6. Neither check proves native runtime behavior or an older GLIBC baseline. Docker ARM64 execution on the current x86-64 host failed with Exec format error; it was not reported as native ARM64 validation.

Linux x86-64 validation passed formatting, all-target/all-feature clippy, 413 library tests, 2 CLI unit tests, 5 architecture tests, and the chunker, description, documentation, xxHash and doctest suites. The initial slim-container contract run had 139 passes and two failures caused by missing pgrep and Python. With procps, Python and the hash-pinned protocol peers installed, a normal container init for child reaping and RCP_REQUIRE_KDC=1, all 141 contract tests passed, including real Kerberos, signed SMB, SFTP, local copy/verify/sync/deletion and MCP child cleanup. These results are native Linux x86-64 container tests, not ARM64 or Windows tests.

.github/workflows/portability.yml is manual and read-only. Its ARM64 job asserts the native architecture, installs the same KDC and protocol peers as existing CI, requires the KDC tests, tests default/TLS/minimal builds and produces a local candidate tarball and checksum without uploading or releasing it. The default and TLS suites include copy/verify, sync/deletion, MCP and job cancellation. The Windows job runs the separate native NTFS qualification script and creates a ZIP/checksum, retained as workflow evidence for seven days. It does not report the Unix process/protocol contract suite as a Windows qualification.

GitHub provides ubuntu-24.04-arm as a standard runner for public repositories. The project uses no self-hosted runners; do not add a paid runner or persistent host access to complete validation. See the GitHub runner reference.

Candidate bundles

Run scripts/test-windows.ps1 -BuildCandidate on native Windows with Rust 1.87 and the existing MSVC toolchain, or dispatch portability.yml with scope=windows-local. The candidate profile uses -C target-feature=+crt-static throughout qualification and packaging, avoiding a separately installed Visual C++ runtime prerequisite. PE import names are recorded in the manifest, and packaging rejects dynamic MSVC runtime imports. The Windows ZIP contains the executable, Apache-2.0 project license, dependency notices, runtime limitations, this support matrix, source/feature/binary manifest and checksums. It remains unsigned and requires the qualified Windows/NTFS profile.

The existing GNU x86-64 Container workflow now packages its exact ABI-gated and native UBI9-smoked executable into a tarball with the same documentation, manifest and checksums, plus its GLIBC gate evidence. Both bundles are retained as workflow artifacts for seven days; neither is a release asset. python -m unittest discover -s scripts -p test_package_candidate.py verifies archive contents, deterministic serialization and refusal of incorrect target, feature, source revision, license/notices and GLIBC evidence inputs.

A streaming experiment was rejected and closed unmerged; its measured regressions stay out of the tree. Unsupported Windows protocols/process features, real Windows SMB peer qualification and full native ARM64 protocol/KDC qualification remain separate work.

Releases and provenance

The Release workflow runs on a vX.Y.Z tag matching the crate version. It runs the contract tests, including mandatory real-KDC Kerberos checks, then builds an x86-64 GNU/Linux binary with the rustls feature for TLS support. The binary dynamically links system GNU libc; Kerberos also needs the host’s GSSAPI library.

The GitHub release contains the tarball, SHA256SUMS, and an SPDX bill of materials named rcp-vX.Y.Z-x86_64-unknown-linux-gnu.tar.gz.spdx.json. The archive includes the binary, LICENSE, README.md, RUNTIME.txt with the required glibc symbol versions, THIRD_PARTY_LICENSES.txt with dependency notices, and the same bill of materials as SBOM.spdx.json. Download the assets from the releases page, verify the checksums, extract the archive, and check its runtime requirements before installing:

sha256sum --check SHA256SUMS
tar -xzf rcp-vX.Y.Z-x86_64-unknown-linux-gnu.tar.gz
cat rcp-vX.Y.Z-x86_64-unknown-linux-gnu/RUNTIME.txt
install -m 755 rcp-vX.Y.Z-x86_64-unknown-linux-gnu/rcp "$HOME/.local/bin/rcp"

Create $HOME/.local/bin first if needed and ensure it is on PATH.

scripts/release.sh (see the repository’s CONTRIBUTING.md) prepares the version bump. Running the workflow by hand on main with a version checks the release, creates the tag, signs it, and publishes in one run.

Signatures and provenance

Every release is signed keylessly through Sigstore from the release workflow’s GitHub identity:

• rcp-vX.Y.Z-x86_64-unknown-linux-gnu.tar.gz.sigstore.json is the cosign signature bundle for the archive.
• rcp.intoto.jsonl is the SLSA build provenance statement for the archive and its SBOM, and rcp.sigstore.json is the Sigstore bundle that signs it. The same attestation is stored with GitHub.

Use cosign v3 or later for every cosign command on this page. The release workflow signs with cosign v3 and its newer Sigstore bundle format. cosign v2.6.1 still verifies the archive bundle below, but cosign v2.x reports “no signatures found” for the images (see Container image and registry artifacts).

cosign verify-blob rcp-vX.Y.Z-x86_64-unknown-linux-gnu.tar.gz \
  --bundle rcp-vX.Y.Z-x86_64-unknown-linux-gnu.tar.gz.sigstore.json \
  --certificate-identity-regexp '^https://github.com/rust-cp/rcp/\.github/workflows/release\.yml@' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com
gh attestation verify rcp-vX.Y.Z-x86_64-unknown-linux-gnu.tar.gz --repo rust-cp/rcp

Signed tags

Tags made by the release workflow (from 0.1.2 on) are signed keylessly with gitsign under the workflow’s identity on main. To verify one, with gitsign installed:

git fetch --tags https://github.com/rust-cp/rcp
gitsign verify-tag vX.Y.Z \
  --certificate-identity https://github.com/rust-cp/rcp/.github/workflows/release.yml@refs/heads/main \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

GitHub’s web interface shows these tags as unverified, because it does not check Sigstore signatures; gitsign verify-tag is the check. v0.1.1 predates this and is an unsigned annotated tag; its release assets are signed as described above.

Container image and registry artifacts

The workflow pushes the hardened runtime image (see container/README.md) to ghcr.io/rust-cp/rcp:X.Y.Z. The image digest is signed with cosign and carries an SPDX SBOM attestation and a build provenance attestation. Verifying them needs cosign v3 or later. The release workflow signs with cosign v3, which stores signatures and attestations in the newer Sigstore bundle format, attached to the image as OCI referrers. cosign v2.x looks only for the older signature tags and reports “no signatures found” for every published image, 0.1.1 and 0.1.2 included.

cosign verify ghcr.io/rust-cp/rcp:X.Y.Z \
  --certificate-identity-regexp '^https://github.com/rust-cp/rcp/\.github/workflows/release\.yml@' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com
cosign verify-attestation --type spdxjson ghcr.io/rust-cp/rcp:X.Y.Z \
  --certificate-identity-regexp '^https://github.com/rust-cp/rcp/\.github/workflows/release\.yml@' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com
gh attestation verify oci://ghcr.io/rust-cp/rcp:X.Y.Z --repo rust-cp/rcp

The bare binary is also pushed as an OCI artifact to ghcr.io/rust-cp/rcp-binary:X.Y.Z and signed the same way. scripts/publish.sh [REGISTRY/REPO] [TAG] performs that push with oras: one layer for the binary, one for SHA256SUMS, and manifest annotations recording the repository, commit, dirty-tree flag, rustc version, target, build time and binary SHA-256.

Both packages are public, so no login is needed to pull them.

oras pull ghcr.io/rust-cp/rcp-binary@sha256:<digest>
sha256sum --check SHA256SUMS
oras manifest fetch ghcr.io/rust-cp/rcp-binary@sha256:<digest> | jq .annotations

Telemetry

None: rcp reports nothing about itself to anyone. What it does connect to is worth stating precisely, because the short answer – “only to the NFS server or S3 endpoint named on its command line” – is wrong: there are six other transports, and hosts beyond those.

rcp opens sockets to:

• the endpoints named on its command line – NFS, S3, HDFS, OCI, WebDAV, ssh://, sftp://;
• with -nodes, every worker host it names, over ssh, whether or not any endpoint is an ssh:// one;
• the hosts those endpoints hand it on to, which is a real hand-off rather than a detail: an OCI upload Location, an OCI bearer realm, and a WebHDFS 307 to a datanode are all hosts the remote chooses. On a railed run -allow bounds them, as the registry and WebHDFS sections above set out;
• the proxy the environment names for any of the above.

It reads: the catalog location; the AWS credential and config files for S3; the registry login files in the order the container tools read them – REGISTRY_AUTH_FILE, $XDG_RUNTIME_DIR/containers/auth.json, $DOCKER_CONFIG/config.json, ~/.docker/config.json; the local passwd and group files to map owners by name; /proc/self/mountinfo for the mount table; the TLS trust store for https – SSL_CERT_FILE, SSL_CERT_DIR, AWS_CA_BUNDLE, and the system bundle when none of those is set; and the files named by -reference, -umap, -gmap and -copybatch when a command takes them; whatever ssh itself reads for ssh://, sftp:// and -nodes; and, for Kerberos, whatever the host’s GSSAPI library reads (krb5.conf, the ticket cache, a keytab) – and that library then dials the KDC, a host on no command line, which is a socket easy to leave off a list like this one.

Security requirements and assurance case

This chapter states what users can and cannot expect from rcp in terms of security, and argues why those expectations are met. Reporting a vulnerability, and how reports are handled, is in the repository’s SECURITY.md.

What rcp is

A command-line program that copies, verifies, resumes, syncs and scans file trees between a local filesystem and network storage (NFS, S3, SMB, SFTP, ssh:// and rcp:// peers, WebDAV, HDFS, HTTPS sources, OCI registries) and tar archives. It runs with the privileges of the user who starts it, keeps a catalog of what it transferred, and sends no telemetry (Telemetry). rcp serve exposes a directory to another rcp, and rcp mcp lets an agent drive rcp within operator-set limits (Driving rcp from an agent).

Security requirements

What you can expect:

1. Integrity of transferred data. A copy hashes every file (SHA-256 by default, BLAKE3 on request) and records it in the catalog; verify compares content, not just sizes and times, and a run that cannot confirm data arrived intact does not report success.
2. Writes stay where they were sent. rcp writes, removes or changes only what is under the destination it was given. Local opens that must not follow a symbolic link use O_NOFOLLOW/O_DIRECTORY, and names that would leave a tree (., .., separators, absolute paths, device names on Windows) are refused.
3. Credentials go only to the endpoint they belong to. Credentials are never taken from a URL (an endpoint carrying one is refused), are redacted from logged and described command lines, and are relayed to helper processes through their environment, not their command line.
4. Verified TLS by default. With TLS (--features rustls), certificates are verified against the system bundle or the bundle the environment names, with TLS 1.2 or 1.3 only. Turning verification off takes an explicit -*.noverify option and prints a warning.
5. Bounded behavior against hostile peers and inputs. A server, peer or file cannot make rcp allocate or loop without bound: lengths and counts from the wire are checked before allocation, and parsers refuse malformed input by name.
6. A network listener is not exposed by accident. rcp serve -listen refuses to start without -allow roots, serves only paths inside them, and binds only loopback unless mutual TLS (-tls.clientca) is configured.
7. An agent stays inside its rails. rcp mcp will not start without -allow, and -readonly and plan confirmation limit what a caller can do.
8. No telemetry. rcp opens sockets only to the hosts listed in Telemetry.

What you cannot expect:

• Confidentiality or integrity on the wire for protocols that have none: plain NFS (-nfs.sec sys), SMB without signing, http://, rcp:// without TLS, and S3, WebDAV or OCI over http. rcp uses these only when you name them; use krb5i/krb5p, -smb.sign, https, davs, rcps or ssh-based transports where the network is not trusted.
• Protection from a malicious server you chose to trust: it can serve wrong data, which verify against a trusted copy detects but cannot prevent.
• Protection from other local users with your privileges, or from root on the host.
• Protection of credentials you pass as command-line options from other local users who can read the process table. Use the environment or credential files instead.
• FIPS validation. rcp is not a FIPS-validated module.

Threat model

Assets: the data being moved and the destination’s existing contents; credentials for the endpoints; the catalog; the host account rcp runs as.

Adversaries considered:

• A network attacker between rcp and an endpoint (reads or alters traffic).
• A malicious or compromised server or peer (NFS, S3, SMB, WebDAV, HDFS, OCI, HTTP, an rcp serve), which can send malformed or oversized responses, names that try to escape the tree, redirects to other hosts, or wrong data.
• A malicious file or archive as a source (a tar, a compressed stream, JSON).
• An agent or automation driving rcp mcp that tries to act outside its rails.
• A remote client of rcp serve -listen.

Out of scope: an attacker who already controls the account rcp runs as, the operating system, or the build and release infrastructure beyond what signed releases and provenance let users check.

Trust boundaries

Secure design principles applied

• Fail-safe defaults. TLS verifies by default; a listener binds loopback only; rcp mcp refuses to start without -allow; credentials in URLs are refused; an endpoint left out of a build is refused by name rather than guessed at.
• Complete mediation. Every path a client or agent names is checked against the allowed roots on each request, not once per session (src/remote/server.rs, src/mcp.rs).
• Least privilege. rcp needs no elevated privileges and asks for none; the catalog is created private to the user (mode 0600 files, private directories, src/catalog.rs); the container image has no shell or package manager (container/README.md).
• Economy of mechanism. One Tree interface for every endpoint and a core that does no I/O of its own (Architecture); tests/hexagon.rs fails if the core reaches an adapter directly. unsafe is confined to the system calls a copy needs.
• Separation of privilege and open design. Integrity is checked by content hash end to end, independently of the transport; all algorithms and protocols are public.
• Least common mechanism. Credentials are scoped to their endpoint kind and are not shared across endpoints or relayed on command lines.
• Psychological acceptability. Refusals name what was refused and why; unsafe options (-*.noverify, -*.insecure) must be named explicitly, and skipping certificate verification prints a warning.

Common implementation weaknesses countered

Evidence that keeps the argument true

Every pull request runs formatting, Clippy with warnings denied, the test suite with real peers, the interop suite against real servers, fuzzing, CodeQL, cargo deny, cargo vet, zizmor, actionlint and gitleaks. The assurance workflow measures line coverage (at least 80%) and checks that the release build reproduces bit for bit. A change that weakens one of the requirements above is a vulnerability under SECURITY.md’s scope.

External dependencies

Where they are listed

Every external dependency is declared in a machine-readable file:

• Rust crates: Cargo.toml, with exact versions and checksums in Cargo.lock.
• Python test peers: tests/support/requirements.txt, hash-locked and generated from tests/support/requirements.in.
• CI tools: scripts/install-ci-tool.sh, each with a pinned version and SHA-256.
• GitHub Actions: pinned to commit SHAs in .github/workflows/.
• Container base images: digest-pinned in container/ and the workflows.
• The released binary’s runtime dependency graph: the SPDX SBOM published with each release (*.tar.gz.spdx.json, and rcp.spdx.json in the repository).

Policy

• Few, and allow-listed. rcp depends on a small set of widely used crates chosen for one job each (TLS, and standard cryptographic primitives: RustCrypto sha2, hmac, md4, md-5, aes and cmac, and the official blake3). The default build’s complete list, support crates included, is scripts/allowed-dependencies.txt, and CI’s dependency-allowlist job fails on any crate not in it. Adding a crate needs the maintainer’s approval in the pull request that adds it, with the reason, and the crate must pass the checks below.
• Licenses. deny.toml allows only Apache-2.0, BSD-3-Clause, ISC and MIT; anything else fails cargo deny.
• Sources. Crates come only from crates.io (deny.toml [sources]); git and unknown registries are refused.
• Audits. cargo vet (supply-chain/) runs on every pull request: each crate version must be audited, covered by an imported audit, or listed as an exemption in supply-chain/config.toml, so a new or changed crate cannot arrive unreviewed.
• Known vulnerabilities. cargo deny checks the RustSec advisory database on every pull request; GitHub Dependabot alerts and security updates are on. A reported vulnerability in a dependency is fixed by updating it, or documented as not exploitable in rcp.
• Updates. renovate.json configures Renovate to propose updates for crates, actions, test peers, images and tools. Updates are not merged automatically, and a version bump of a pinned tool must carry its independently verified checksum.

The crates are used through Cargo, not vendored or forked, so a fixed version reaches rcp with a Cargo.lock update.