Archive — history, not state. Kept for its reasoning and its evidence; its plan is closed.

Battery speed — where the test battery’s seconds actually go

Branch cp/battery-speed, worktree .claude/worktrees/night-battery, base dev@7c02555dd. Measured 2026-08-04 on darwin/arm64 after rm -rf .pit build cold-build pit && make (the C build itself is 22.3s and is not counted below).

Measurement hygiene warning. Throughout this session another agent’s worktree (night-battery’s neighbour night-seedspeed) was running pit bootstrap at 1205% CPU with a system load average of 12. Absolute wall numbers below carry ±40% run-to-run noise; every conclusion here rests on ratios measured inside one run (the harness’s own realize/deadline split, and the pit.compile.plan.telemetry@1 counters), which are immune to that noise. Where a wall number is load-bearing the best of ≥2 runs is quoted.

1. Attribution table

Battery components

Realize vs execute — the headline ratio

The harness already measures both halves and now prints them. Raw rows:

# ./pit test suite, cold
slowest  class  deadline  realize  total
suite    actor  2.3 us    59.4 s   59.4 s

# ./pit test suite, warm
suite    actor  2.2 us    92.7 ms  92.7 ms

# ./pit test run tests/vm_suite.ce, warm
vm_suite actor  7.5 us    53 ms    53 ms

475 assertions execute in 2.3 microseconds. Realizing the actor that runs them takes 59.4 seconds. That is a ratio of 2.6 × 10^7 : 1. The vm suite is the same shape (7.5 µs of test, 53 ms–14 s of realize). This is the finding John’s instinct was pointing at, and it is not a defect — it is what “the compiler is written in pit and the test harness realizes each test file as its own actor” costs.

Individual heavy files

tests/compile.cm’s 1.54 s warm number is genuine test work (it compiles snippets); the other 22.9 s is one-time module realization.

Daemon / client floor

Daemon overhead is not a factor. 20 ms per command against a ~50 s battery is 0.04%. The interesting floor is the third block: even with every module artifact already in the object store, standing up one novel compiler-importing actor costs 1.6 s of pure closure-walk-and-link.

Realization waste across test files — telemetry

97 pit.compile.plan.telemetry@1 blocks captured across one full ./pit test (~100 baseline files), aggregated:

Answer to “does each test file rebuild shared deps”: no. Zero. Every one of the 138 mcode misses is a distinct derivation key; the store serves 2518 hits. The cross-file cache is working exactly as designed.

But note the second-largest row: graph_walk_elapsed_s (57.8 s) is 1.7× the actual compile time (34.0 s), and it is not compilation. It is fetch_closure re-walking the module graph per test file: read every source file, blake2 it, compute mcode_cache_key, look up the artifact, resolve every import through the resolver, and stat it. Warm, on a 38-module closure, that is ~1.0 s per test file for work whose answer was identical the previous 96 times.

Fuzz per-program cost breakdown

Measured over the same 20 generated programs (seeds 1000–1019), instrumented at the arm boundary; order-independent (verified by running the arms in reverse):

Fuzz is 99.99% compilation and 0.006% execution. The linked arm does not re-link shared prelude work per program — run_fully_linked links exactly one unit with no imports, and semantic_link_shapes() (the only multi-unit link) is gated to i == 0. That part is already tight.

Ship-vs-dev profile — where it is chosen

PASSFLAGS.resolve defaults to profile: "ship", stage: "unit" (pit-compiler/passflags.cm:192-193), and every shop lane hard-defaults compiler_profile to "ship" (shop_build.cm:74, shop_fetch.cm:440, build_fleet.cm:69,82, builder_worker.ce:76,136, shop_store.cm:103, derivation_status.cm:38). A test actor’s realize context confirms it:

"mode":"dev", "compiler_profile":"ship", "payload_mode":"mach"

Measured cost of that choice on four real fleet modules (shop_build.cm, streamline.cm, shop_fetch.cm, test.ce) through mcode_unit_result:

ship = 17.46 s    dev = 12.39 s    ratio = 1.41

So tests are realized ~41% more expensively than dev would cost. See proposal P3 for why flipping it is not a free win.

Serialization — the tester actor and the runtime

shop_tools/test.ce:47-52 already has the lever:

def TEST_SPAWN_CONCURRENCY = ... os_ref.getenv("PIT_TEST_CONCURRENCY") ... default 1

spawn_queued_tests() (test.ce:664-672) drains queued_actor_tests until length(pending_actor_tests) == TEST_SPAWN_CONCURRENCY. Shipped default is 1, so test files run strictly one at a time. The fuzzer is likewise a strict sequential step() chain (fuzz.ce:98-138), one program per turn via $delay(step, 0) (measured: $delay(f, 0) costs 12 µs, so the chain itself is free).

But raising either number buys nothing on this machine. Sampling the daemon during a fuzz run:

%CPU: 99.3  99.3  99.5  99.5  99.8  99.8  97.3  97.3

The pit node never exceeds one core. grep -rl pthread_create source/ returns exactly one file, and it is pgo_collector_test.c — the runtime has no worker threads by construction. The build fleet’s own telemetry agrees: fleet_peak: 1, compile_peak: 1, fleet_global_peak: 1 on a 33-module cold realize with parallel(jobs, null) (unthrottled). Actor concurrency inside one node interleaves turns; it does not add cores. Fanning the battery out means more processes, not more actors — which is a design decision, not a tuning knob.

2. Findings ranked by wasted seconds

F1 — Closure re-walk: ~58 s per full pit test, ~2 s per battery run

fetch_closure (pit-shop/shop_fetch.cm:43) resolves the whole import closure from scratch for every realization. Per file it reads the source, hashes it, computes the derivation key, probes the artifact store, resolves every use() through the resolver and astats the file (analyze_locator, lines 394–486). Nothing about that answer changes between test file 3 and test file 97 for the ~35 shared fleet modules, yet it is recomputed 97 times.

Evidence: Σ graph_walk_elapsed_s = 57.8 s vs Σ compile_elapsed_s = 34.0 s over one ./pit test; and the warm-store realize floors (0.23 s trivial → 1.63 s compiler-importing) which contain no compilation at all.

Cost inside the specified battery is small (three realizations), but this is the dominant non-compile cost of any multi-file test run and of make check.

F2 — join_types is 46% of fuzz wall time and is off in every shipped profile

The fuzzer’s third dimension compiles each program with join_types: true. Isolated measurement (arms reordered to rule out warm-up effects, two runs, stable to 2%):

streamline WITHOUT join_types : 1.74 - 0.24 = 1.50 s / 20 programs
streamline WITH    join_types : 3.10 - 0.24 = 2.87 s / 20 programs
                                              -> +91%

join_types alone costs more than every other streamline pass combined. And pit-compiler/passflags.cm:86-97 sets join_types: false in all three shipped profiles (dev, dev/link, ship, small) — the header comment says so explicitly: “its current ship/small default is off because the corpus compile-cost measurement did not justify enabling it.”

So ~46% of every pit fuzz — ~21 s of a 47 s default run, ~8 s of the 17 s 40-program gate — differentially tests a pass that no shipped configuration enables. That is a deliberate coverage choice (the pass exists and must not rot), but it is by far the largest single line item in the warm battery and it should be a conscious choice. See P2.

F3 — Ship profile on test realization: ~29% of cold realize time

Cold ./pit test suite spends 49.8 s of its 59.4 s in compile_elapsed_s. Measured ship:dev ratio on real modules is 1.41, so ~14 s of that 50 s is ship-only optimization applied to code whose measured execution time is 2.3 microseconds. See P3 for why this is not a free switch.

F4 — The battery is single-core on a ≥12-core machine

Confirmed by direct sampling (99% CPU ceiling) and by absence of pthread_create in source/. Every second in the table above is one core’s second. This is the documented actor model, not a bug — but it caps every other fix: the cold battery cannot go below “compile 33 pit modules serially”.

F5 — Non-findings worth recording (so nobody re-measures them)

• No cross-file re-realization waste. 138 misses, 138 distinct keys, 0 repeats.
• Daemon/client overhead is 20 ms, 0.04% of the battery. Not worth touching.
• The linked fuzz arm does not re-link shared prelude work. link_shapes is already gated to the first program.
• $delay(f, 0) costs 12 µs. The fuzzer’s per-program turn boundary is free.
• mcode_unit_result re-parsing in the linked arm is real but tiny: the whole parse+analysis is 3.4 ms of that arm’s 79 ms (≈4% of the arm, ≈1% of fuzz).
• The -e: eval executables rebuilt on each pit test cost 0.06–0.12 s each with 0–1 compile misses. Cosmetic, not a cache defect worth chasing.

F6 — Out of scope but noticed: pit test (full baseline) is red on this base

dev@7c02555dd has 6 pre-existing failures in ./pit test (mach_pool_emit ×2, heap timeout, network_info / recipes compile failures, shop_resolve). The heap timeout alone burns 30 s of the 55 s warm pit test run waiting out its deadline — i.e. on this base, more than half of a warm full-baseline run is spent waiting for a test that is already failing. None of these are in the three-component battery, which is green.

3. Changes made — FOR REVIEW

None.

This is a deliberate result, not a shortfall. The brief asked for “clearly-safe mechanical fixes — a redundant recompile, an obviously wrong cache key, a serial loop the design already permits to be concurrent, a profile default misapplied.” Each of those four was looked for specifically and each was measured:

• redundant recompile — 138 misses, 138 distinct keys, zero repeats (F5).
• wrong cache key — the store serves 2518 hits against 138 misses; the keys are correct.
• serial loop the design permits to be concurrent — the loop exists (TEST_SPAWN_CONCURRENCY, and the fuzzer’s step chain) and the design does permit N, but the runtime is single-threaded (F4), so raising it converts wall-clock into interleaving with no gain and non-trivial risk to the harness’s per-test deadline accounting. The comment at test.ce:41-46 says the parallel- safety question “can be ASKED without a rebuild” — this report asks it and answers not worth it on one core.
• profile default misapplied — the ship default is applied uniformly and deliberately (F3); changing it for the test lane forks the artifact cache (P3).

Every remaining candidate touches design or semantics. They are listed below instead of implemented.

4. Proposals — evidence attached, not implemented

P1 — Session-scoped closure cache in fetch_closure (largest win, ~50 s/make check)

Evidence: Σ graph_walk_elapsed_s 57.8 s vs Σ compile_elapsed_s 34.0 s over one ./pit test; 1.0 s of pure closure walk on a fully-cached 38-module graph.

Shape: memoize the per-locator result of analyze_locator (shop_fetch.cm:394) — {metadata, edges, executable_edges, source_dep} — keyed on (locator, source stat identity), for the lifetime of one daemon generation. The freshness inputs are already computed in that function (linked_upstream_stat, pitfs.astat), so the invalidation predicate exists; what is missing is the map.

Why it is a proposal and not a fix: the closure walk is also the freshness gate. package_freshness_started already encodes a once-per-closure rule; a once-per-session rule is a different trust statement about linked packages changing under a running daemon, and that is a ruling, not a refactor.

P2 — Make the fuzzer’s join_types dimension explicit and samplable

Evidence: F2 — 46% of fuzz wall time, +91% on streamline, a pass that is off in dev, ship and small.

Shape: either (a) run the join_types arm on every Nth program rather than every program (the same treatment link_shapes already gets at fuzz.ce:107), or (b) put it behind a flag that pit fuzz sets by default and CI sets always. Option (a) at N=4 would take pit fuzz from ~47 s to ~31 s and the 40-program gate from 17 s to ~11 s while still sampling ~25 programs per 100 through the pass.

Why it is a proposal: thinning a differential dimension is a coverage decision. The pass is a live compiler pass; a fuzz arm that only sometimes runs finds bugs later. That trade is John’s to make, not a mechanical fix.

P3 — Realize the test lane under dev (measured 1.41× cheaper) — with a caveat that may kill it

Evidence: F3 — ship:dev = 1.41 on real fleet modules; ~14 s of the 50 s cold test suite compile.

The caveat, stated plainly: compiler_profile is folded into shop_store.mcode_cache_key (shop_store.cm:103). A daemon that realizes test actors under dev and everything else under ship compiles the entire shared fleet graph twice and shares nothing between the two. On a warm developer daemon that is a large net loss, not a win. It only pays if the whole dev tree moves to dev — which contradicts the tests that specifically assert ship output (tests/store_freshness.ce pins both profiles by name; tests/compiler_profile.ce is about profile selection itself; diff_runner’s SHIP_PLAN and the fuzz join arm both name ship explicitly). Recommend: do not pursue without a whole-tree ruling.

P4 — Process-level fan-out for the battery (the only route past F4)

Evidence: F4 — 99% CPU ceiling, no pthread_create in source/, ≥12 cores idle.

Shape: the battery’s three components are independent and could run as three processes; pit fuzz could shard [start_seed, start_seed+N) across K processes and merge summaries. Nothing in the fuzzer’s contract is order-dependent — it is a pure function of seed.

Why it is a proposal: this is a harness architecture change (who owns the shop, how failures merge, how gate.sh rules), and it interacts with the store’s concurrent-writer story. Large win available (theoretically 4–8× on fuzz), large design surface.

P5 — Low-value cleanups, listed so they are not re-discovered

• diff_runner.arun_source parses each fuzz program twice: once via compiler.analyze (shared by three arms) and once inside compiler.mcode_unit_result(src, …) for the linked arm. The compiler already has the *_from(parsed, opts) idiom (analyze_result_from, compile_result_from); an mcode_unit_result_from would close it. Measured worth: ~1% of fuzz. Not worth the new export on its own — fold it in if that seam is opened anyway.
• shop_tools/test.ce runs acollect_tests_with_ext twice per invocation (once for .cm, once for .ce), each doing its own pkg.resolve_package + pkg.list_files with glob tests/**/*. One walk collecting both extensions would halve it. Worth tens of milliseconds.
• build_telemetry.finish (build_telemetry.cm:342) JSON-encodes and logs compile_keys, mcode_encode_keys, mach_lower_keys, k_mcode_miss_keys and k_mach_miss_keys — five maps holding largely the same hashes — on the compile channel for every realization. ~13 KB of JSON per cold test-file realize. Real but sub-1% work; and tests/build_telemetry.cm pins the accounting, so trimming it is a contract change.

5. Gates

Battery green on this branch (nothing was changed, so this is a base-state confirmation, not a before/after):

check: vm suite OK                                    0.13 s
./pit test suite    passed: 475  failed: 0            0.13 s
./pit fuzz 40       failed: 0  (2 arms x ~300 checks) 17.10 s

Cold battery (after rm -rf .pit build cold-build pit && make): 120.9 s. Warm battery: ≈ 45–51 s at the default 100 fuzz programs, 17.4 s at the 40-program gate.

6. The one-line answer

The battery is not slow at running tests — it runs 3782 assertions in single-digit milliseconds. It is slow at building the things that run them: 99.99% of pit fuzz and 99.9999% of pit test suite is the pit compiler compiling pit, once per test file and four times per fuzz program, on one core. The caches are correct and there is no redundant work to delete. Every remaining second is bought back only by compiling less (P2, P3), walking the graph less (P1), or using more cores (P4) — all four of which are rulings, not repairs.

7. Closure cache + fuzz sampling — P1 and P2, implemented

Branch cp/closure-cache, worktree .claude/worktrees/night-closurecache, base lane/cp@e28b7a995 (which carries the warm-staleness fix — the memo below is only safe on top of it). Measured 2026-08-05 on darwin/arm64.

Measurement hygiene, again. Five other agents’ worktrees were compiling throughout; system load average ran 17–24 on this machine. Every wall-clock number below is best-of-N and carries ±30% run-to-run noise. The load-bearing measurement in §7.1 is deliberately built to be immune to it: both arms run inside one process, interleaved with the same neighbours.

7.1 Fix 1 — the session closure memo (P1)

What it caches, and what it deliberately does not

fetch_closure re-resolved the whole module graph from scratch for every realization. Per member: read the source, hash it, compute the derivation key, probe the artifact store, resolve every use() through the resolver’s candidate rings, and astat the file. For the ~35 shared fleet modules that answer is identical for every test actor in a run.

The memo caches the resolution graph of one member — which locators its imports and claims resolve to, which natives and executables it pulls in. It caches neither the bytes nor the metadata, and both exclusions are load-bearing rather than pedantic.

The two rules, as implemented

Rule 1 — never skip file(). shop_source.file is the freshness point since the warm-staleness fix, and it runs for every member of every walk, memo or not. The memo is consulted strictly behind it, and the key is shop_store.mcode_cache_key — a hash of the bytes file() just returned, plus the kind, numrep, link mode, compiler identity, compiler profile and streamline_disable. A hit is therefore a pure function of source text that has already been verified against its upstream. A memo in front of file() would re-open the staleness defect at closure scale, which is exactly what staleness.md §5 refused.

Rule 2 — a closure is a graph, not a file. Three invalidation axes:

Only fully-resolved walks are stored. An unresolved import, a claim error, a missing or mounted executable, or a scan error blocks the memo — those answers can change when a package is fetched later in the same session, and a memo that replays a failure would pin it for the daemon’s lifetime.

A null endowment answer does not block, and this was the difference between a memo that works and one that does not. claim_names_of_unit raises a claim for every language global a unit touches — length, text, arrfor — and the raw runtime provider answers almost all of them with no unit at all. Treating a null endowment answer as a failure refused 20 of the 29 members of shop_tools::test, and the measured win was zero. An endowment that resolved to an error still blocks.

The memo is process-lifetime, like shop_source’s identity map: empty at every daemon start, so it can never let a daemon inherit a belief from a previous process. It is bounded at 4096 entries by a whole-map clear (the cheapest bound is the right one for a pure speed-up over a correct walk).

The defect this fix introduced, and its repro

The first version cached metadata along with the resolution and replayed it. That is wrong, and it is worth recording because it is invisible on a warm store:

The first walk of a member that has never been compiled carries only scan-derived metadata. The walk after its derivation must report the stored slim metadata with its content hash — that is what lets assembly and the executable manifest consume the walk instead of re-deriving. Replaying the memo’s copy pinned every member to its pre-derivation metadata for the whole daemon session.

It showed up only on a cold ./pit test (closure_checks: “warm closure did not consume stored slim metadata for runtime/tests/fixtures/lto_mod.cm”, +2 failures). The fix: probe the artifact on every walk, memo or not, and have a hit report this walk’s metadata. The memo caches resolution; metadata is re-read.

The cheap repro, without a cold rebuild — append a comment to tests/fixtures/lto_mod.cm so its derivation key is one no artifact exists for, then ./pit test run tests/store_freshness.ce. Pre-fix: fails. Post-fix: passes.

Counters

shop_build.closure_memo_stats(), in the shop_source.source_freshness() style — process-lifetime totals a test can assert on without a clock:

{hits, misses, revalidations, invalidations, stores, clears}

revalidations counts the rule-2 stat per candidate hit; invalidations counts the ones where the identity had moved and the member was re-walked.

Regression test

tests/store_freshness.ce → closure_memo_checks, run after linked_read_freshness_checks. Probe files are written into the runtime package’s fixtures directory, dot-prefixed so the test collectors’ tests/** walk skips them, and removed on every exit path. The probe module carries a $self_id token so its derivation key is one no previous run published an artifact for — without it the store satisfies assertion 3 vacuously on a rerun.

All counter-based:

1. accounting identity, asserted on every walk: hits + misses equals the closure’s member count, so a miss cannot hide.
2. repeat walk of an unchanged program answers every member from the memo (misses == 0).
3. after derivation, the next walk reports the newly stored slim metadata (schema == pit.mcode.unit.metadata@1, content hash present) — the assertion that catches the defect above. Verified load-bearing by reintroducing the bug: it fails, and passes again when reverted.
4. edit one member → exactly that member re-walks (misses == 1), the rest still hit.
5. add an import to a memoized member → the new module appears in the closure, which it cannot if the memo answered from the old edge list.

Measurement

The honest one, built so machine load cancels: walk the closure of every .ce test program (213 programs, 4552 member walks) twice inside one process — once resetting the memo before each walk (the pre-fix cost), once with it live.

13.4×, 300 s of closure walking removed from the work fetch_closure does across the test corpus. invalidations = 0 across the run, i.e. no member’s stat identity moved under a walk that had its bytes.

Where that lands in a ./pit test wall — and where it does not

This is far short of the ~50 s the brief expected, and the reason is a correction to F1’s attribution rather than a shortfall in the fix.

compile_graph_root has two lanes. A program addressed by locator goes through fetch_closure. A program supplied as inline source — spec.source != null — skips fetch_closure entirely and recurses through compile_graph_unit. telemetry.graph_done wraps both, so on the eval lane graph_walk_elapsed_s envelops the whole recursive compile rather than a closure walk. shop_tools/test.ce spawns .cm module tests as inline wrapper source (build_module_test_wrapper) and only .ce actor tests by locator. So a large share of the measured 57.8 s was never fetch_closure at all.

The second reason is bigger: on a warm store most test actors hit a validated realize entry, and fetch_closure never runs. The closure walk is paid on realize misses. So the 300 s this fix removes is paid — and saved — exactly in the loop that matters for development: a compiler-touching edit moves the builder identity, misses every realize key by construction (staleness.md §2), and every test actor then walks its closure. It is not visible in a back-to-back warm rerun where nothing changed.

7.2 Fix 2 — sampling the fuzzer’s join_types arm (P2)

shop_tools/fuzz.ce passed join_types: true for every program. It now samples every 4th (JOIN_TYPES_EVERY = 4), the same treatment link_shapes already gets — except link_shapes pins to i == 0 because the multi-unit link it exercises is program-independent, while join_types is program-dependent and so samples across the seed range instead.

Sampled rather than dropped: the pass is live compiler code and must not rot. A 100-program run still differentially checks ~25 join-types compilations; the 60-program gate checks ~15. diff_runner already took join_types as an opt and records dimensions.join_types.enabled per program, adding no counts for a program whose arm did not run — so the dimension totals report the sampled subset, never a silent zero.

./pit fuzz 60 --seed 4242, same store, same session, best of 2:

−29%. Every other dimension is bit-identical; only the sampled arm’s count moves, and by exactly the 1-in-4 the constant says.

7.3 Gates

All on cp/closure-cache, after rm -rf .pit build cold-build pit && make.

7.4 What is still on the table

• F1’s remaining time is on the eval lane, not in fetch_closure. .cm module tests are realized from inline wrapper source and walk their graph through compile_graph_unit’s recursion, which has no memo and re-resolves per test file. That recursion consults state.resolved_files when a closure walk supplied one — which the eval lane never does. Giving the eval lane a preloaded_closure (it has a real root: the single use() the wrapper emits) would put those ~44 test files on the memoized path too. Not attempted here.
• P3 and P4 are untouched and their verdicts in §4 stand.

Source: plans/archive/night-2026-08-04/battery-speed.md