Archive — history, not state. Kept for its reasoning and its evidence; its plan is closed.

Warm-store staleness — a linked package’s snapshot was never the truth

Branch cp/warm-staleness, worktree .claude/worktrees/night-staleness, base dev@0c6f6aacd. Defect class: cache correctness in the shop. A warm .pit store served a compile of source bytes that no longer existed on disk, silently, with every documented freshness contract intact and none of them on the path.

1. The defect, reproduced

Exact repro, run in this worktree:

git checkout 7c02555dd && make                    # cold build, pre-cp2b
./pit test run tests/mcode_link.cm                # 15/15 — warms .pit/sources
git checkout cp/warm-staleness && make            # = dev@0c6f6aacd, exit 0
kill the daemon; rm .pit/pit.pid .pit/pit.sock    # fresh daemon
./pit test run tests/mcode_link.cm                # 13 passed, 3 FAILED

The three failures are the census-reading tests, and they read -1 for every new counter (link_guards_before/after/removed, link_fact_slots, link_record_origins) while still reading correct values for the counters that existed pre-merge (constant_members=1, function_members=1). That is the signature of the pre-merge pit-linker/mcode_link.cm being what ran.

Direct evidence, taken from the store immediately after the failing run — every changed-and-snapshotted file was still at its 7c02555dd bytes:

$ for f in $(cd .pit/sources && find . -type f); do  # compare snapshot vs repo
STALE: pit-compiler/streamline.cm
STALE: pit-compiler/stream_ir.cm
STALE: pit-compiler/compiler.cm
STALE: pit-linker/mcode_link.cm

$ shasum -a 256 pit-linker/mcode_link.cm .pit/sources/pit-linker/mcode_link.cm
5cedef56…  pit-linker/mcode_link.cm            35334 bytes   (0c6f6aacd)
fb604f2c…  .pit/sources/pit-linker/mcode_link.cm  28773 bytes   (7c02555dd)

git diff --name-only 7c02555dd 0c6f6aacd changes exactly four files that live in a snapshotted package. All four were stale. The staleness was total, not partial.

Confirmation of the causal chain (one command, no code change):

# a throwaway probe_linker.ce at the repo root: def linker = use('pit-linker::mcode_link')
$ ./pit probe_linker.ce
$ shasum -a 256 .pit/sources/pit-linker/mcode_link.cm
5cedef56…                                      # snapshot repaired
$ ./pit test run tests/mcode_link.cm
passed: 16   failed: 0

Running any locator-addressed program that imports the package repaired the snapshot and the same eval-addressed test went green. Nothing else changed.

2. Root cause

Layer: pit-shop/shop_source.cm, file() — the source-read seam. Not the realize index, not the realize entry’s dependency validator, not the linked-upstream stat short-circuit. Those were all innocent; none of them ran.

Key: there wasn’t one. .pit/sources is addressed by locator, not by content or by time: source_cache.key(locator) = hash("pit.source.file@1\n" + locator), a catalog entry mapping that name to whatever content hash was last written for it. source_cache.get(locator) returns those bytes. The map has no freshness component of any kind, by construction.

Why the hit passed: nothing asked a question. file()’s own docstring said where freshness was supposed to come from:

D1 marks linked files current once per closure walk; ordinary compiler/realizer reads then use that walk’s cached snapshot.

The closure walk — shop_fetch.analyze_locator — is the only lane that passes upstream_current: is_linked_package(pkg), which is the flag that makes file() skip the snapshot and read the link target. fetch_closure refuses eval programs (request_program_is_eval → "fetch_closure does not support eval scripts"). ./pit test run <file> realizes an eval program (-e:blake2:…), so shop_build.compile_graph_root takes the preloaded_closure == null-but-eval path into compile_graph_unit → compile_unit → shop_source.read_source → locator_blob → file({locator}) — no upstream_current, snapshot served, directory never consulted.

Why the realize cache didn’t save it either. The linked-upstream freshness contract (content compare, stat short-circuit, and John’s first-validation belt-and-suspenders rule) lives in shop_realize.source_dep_valid, and that is reached only while validating a cache hit. validate_realize_entry returns false at the builder-identity check —

if (!record.builder || record.builder.compiler != identities.compiler ||
    record.builder.lowering != identities.lowering) { cb(false); return }

— before a single dep_valid runs. The cp2b merge moved pit-compiler/ and pit-shop/, so the compiler/lowering identities moved, so every realize key missed by construction. A miss performs no freshness check at all: it goes straight to a rebuild, and the rebuild reads the snapshot.

So the two halves compose into the exact failure observed: the one event that guarantees every realize key misses (a compiler-touching merge) is also the one that removes every freshness check from the run. The mcode cache key folds the same stale source text the compile used, so the result is internally consistent and nothing anywhere reports a problem.

Three sentences. .pit/sources is a locator→bytes snapshot with no freshness component; the only thing that ever refreshed it for a linked package was the closure walk’s per-file upstream_current read, and fetch_closure refuses eval programs, so the -e lane (./pit test run) compiled straight out of the snapshot. The realize entry’s linked-upstream contract could not catch it because it only runs while validating a cache hit, and a compiler-touching merge misses every realize key by construction. Result: a fresh daemon on a warm store served a pre-merge compilation of an edited linked package, and make seed could not clear it because the seed press’s own closure never touches pit-linker.

Suspects, ruled out

3. The fix

pit-shop/shop_source.cm — freshness becomes file()’s own job, at the seam that hands bytes to the compiler, instead of a property a caller had to remember to ask for. The ruled contract is unchanged; it is now enforced for every reader rather than for one lane.

Per process, per locator, for a linked package with a directory upstream:

1. No identity recorded in this process → read the upstream. The map is empty at every daemon start, so a daemon starting against a directory edited while it was down cannot inherit a snapshot. This is John’s first-validation belt-and-suspenders ruling (2026-08-04), applied one layer down at the seam that actually reads bytes.
2. Identity recorded and the link target’s identity (size + mtime-ns + inode) still equals it → serve the snapshot. Same stat contract, same accepted blind spot (make’s and cargo’s), no weakening.
3. Anything else — identity moved, or unavailable on this platform → read the upstream.

The identity is taken before the read, never after, matching the rule the realizer already follows: it is older than the bytes being snapshotted, so a write racing the acquisition leaves it mismatched and forces the next read back to the upstream. It is recorded only after source_cache.put succeeds, and cleared when the upstream reports the file gone.

wants_manifest_current is computed from what the caller asked for, before the escalation, so a forced-current file read cannot drag its package manifest current with it — that is what made D1 freshness quadratic in a linked package’s width, and the fix must not reintroduce it.

Cost. Non-eval lanes: unchanged. Their closure walk already asks for current bytes; that read now records the identity, and the compile read that follows short-circuits on one stat instead of the two store reads (catalog_get_value + object_get_value) it used to pay. Eval lanes: one live directory read per linked locator on first touch per daemon session — the cost the non-eval lanes have always paid, and the price of the answer being true. On a platform with no bootstrap_stat_id (console arms, firmware predating it) rule 3 applies every time and linked reads never short-circuit; that is the honest reading of the contract, which says a null identity means “fall back”, never “unchanged”.

New observable, in the freshness_counts style already used by shop_realize: shop_source.source_freshness() → {linked_upstream_reads, linked_snapshot_skips}, process-lifetime totals, for tests to assert on without a clock.

4. The regression test

tests/store_freshness.ce → linked_read_freshness_checks, run right after the existing linked_freshness_checks, in the same probe-file style (dot-prefixed so every package walk’s !.* glob skips it; removed from both the link target and the snapshot on the way out; self-skips with a message on an installed-only shop that has no linked package).

It drives shop_source.file with no upstream_current anywhere — exactly what a compile does — and asserts, all counter-based, none by timing:

1. the first ordinary read of a linked locator in this process goes to the upstream (linked_upstream_reads + 1), whatever the snapshot says;
2. an unchanged file’s next read does not (linked_upstream_reads unchanged, linked_snapshot_skips up);
3. a file edited behind the shop’s back — written straight into the link target with no shop call — is served fresh on the very next ordinary read (bytes assertion + linked_upstream_reads + 1).

Assertion 3 is the regression: before the fix it returned the previous bytes. The package manifest is read once outside the measured window, because it is itself a file in the linked package and would otherwise move the same counters.

5. Verdict on the held battery optimization

(Session-scoped closure cache in shop_fetch.cm:394, ~50 s win, held on this root cause.)

Under the pre-fix contract: NO, and it would have been the same bug at scale. analyze_locator’s file(..., {locator, upstream_current: linked_current}) was the entire freshness mechanism for linked packages. A session-scoped cache that answers before that read removes the last checkpoint, generalizing tonight’s eval-lane defect to every lane in the system.

Under the fixed contract: YES, conditionally. Freshness no longer lives in the walk, so caching the walk no longer removes it — provided the cache is consulted after the upstream-truthful read, never instead of it. Two rules:

1. Never skip the file() call. With the fix, file() is what makes bytes true; a cache placed in front of it re-opens exactly this hole. Placed behind it and keyed on the source hash (which shop_store.mcode_unit_cache_artifact already is), the scan result is a pure function of bytes that have just been verified, and reuse is sound.
2. A closure is a graph, not a file. Its shape — which locators, packages and natives are in it — changes when an edit adds or removes an import, claim or endowment edge. So a closure-level memo must additionally validate, on every hit, that each member’s link-target stat identity still equals the one recorded when the closure was built (shop_source.linked_upstream_stat, N stats, not N reads), and must fold package_roots.links()’s stat key plus the target/profile/link-mode axes into its key — a new dev link can move a package’s root without touching any file in it.

A bare “same program → same closure” memo keyed on the program locator alone is unsafe and should not land. With rules 1 and 2 it is safe, and the stat-based revalidation keeps most of the 50 s.

6. Gates

All run in .claude/worktrees/night-staleness on cp/warm-staleness@5df49d4f3, after rm -rf .pit build cold-build pit && make (exit 0).

The repro, post-fix

Re-run without a rebuild by putting the store back into exactly the failing state — a snapshot holding pre-merge bytes, plus a fresh daemon (the process-lifetime identity map empty, which is the condition rule 1 answers):

git checkout 7c02555dd -- pit-linker/mcode_link.cm tests/mcode_link.cm
kill the daemon; ./pit test run tests/mcode_link.cm     # 15/15, snapshot := fb604f2c (old)
git checkout cp/warm-staleness -- pit-linker/mcode_link.cm tests/mcode_link.cm
kill the daemon; rm .pit/pit.pid .pit/pit.sock          # fresh daemon
./pit test run tests/mcode_link.cm

snapshot BEFORE : fb604f2c…   (pre-merge)
repo file       : 5cedef56…   (merged)

passed: 16   failed: 0
snapshot AFTER  : 5cedef56…   (repaired by the read itself)

Pre-fix, this exact state produced passed: 13 failed: 3. The compile telemetry for the passing run reads k_mcode_hits: 41, k_mcode_misses: 0: the fresh bytes hashed straight onto the already-cached merged compilation, which is the correct outcome — the fix restores truth without costing a recompile.

Adjacent, not fixed (out of scope, noted)

Package resolution still runs off the snapshot: direct_package_root returns source_cache.root(pkg) and deliberately does not fall through to the host directory, so a file newly added to a linked package is not resolvable until something mirrors it. That is a stated property of C4 (“Directory upstreams do not fall through to their host directory”), not a regression, and it is a different question from the one this defect asked — but it is the remaining way a linked package’s contents and the shop’s belief about them can differ.

Source: plans/archive/night-2026-08-04/staleness.md