Archive — history, not state. Kept for its reasoning and its evidence; its plan is closed.

Unified native suspension experiment

Status: completed composed integration candidate.

Original isolated baseline: 1ea3d92a.

Composed performance baseline: 7e3bbc90; checked-array-load composition: a3f7a284.

This is the second native-suspension experiment. It implements the continuation stack proposed by the rejected scalar prototype and completes the cross-engine contract. The goal is correctness under Pit’s actor rule, not a benchmark win: native language code must release its host C stack when the scheduler asks, including while a Pit call chain is in progress. C module activations remain the explicit non-freezable boundary.

No semantic mcode operation, language-visible frame field, or per-call frame word was added. Resume segments, liveness, and polling are QBE lowering facts.

Result

The experiment supports all of these with the same Pit_ResumeVM entry:

• all-native loops, dynamic calls, direct recursion, and depth greater than one;
• Mach calling native and receiving the eventual result;
• native calling Mach, including a Mach callee that itself suspends;
• native -> Mach -> native nesting;
• final disruption through any of those paths;
• a copying GC after every suspension; and
• apply() of a native function (previously an opaque C helper barrier).

I recommend the architecture for integration because native freezeability is a language invariant. The residual cost is the price of making previously unfreezable native execution obey the language, not an optional speed feature. It is materially better than the first prototype on every problematic macro: mandelbrot fell from +15.6% to +1.3%, closure from +9.9% to +3.6%, and the rejected 2x mandelbrot assembly growth fell to +20.8%. Recursive fib remains the outlier at +12.3%; it has no loop backedge, so every useful scheduler observation comes from the shared call counter. That tax is real and should be the next call-ABI optimization target.

Execution model

One lazy continuation stack

NativeRTState owns a LIFO of stable four-entry chunks. An entry contains an embedded PitGCRef, Mach frame-stack mark, program counter, result destination, kind flags, and initialization state. Entries are allocated only when engines cross or an outer Mach state must coexist with a suspended inner state.

The embedded GC root deliberately costs more than a custom scanner. It makes each stable entry independently correct under moving GC and avoids another context-owned scan protocol. Inactive cells are null. Chunks never move while a root is linked. The context destructor deletes every initialized root.

The stack stores only engine boundaries/leaf state; ordinary all-Mach and all-native calls keep their existing frame chains. Depth remains bounded by ctx->stack_limit. Allocation failure or bound failure disrupts before state is discarded; there is no unfreezable fallback.

Stack-flat unwind and resume

Generated native code assigns lowering-only resume IDs. On suspension it spills the union of promoted values that are live at any suspension site, stamps the first unstamped caller, marks native state suspended, and returns the existing internal exception token up the host C stack. A single token retains the old hot exception check; context state distinguishes disruption from scheduling on the cold edge. The public dispatcher maps the internal token back to PIT_SUSPENDED.

Resume enters the retained top native frame. When it completes, the result is delivered to the preceding awaiting entry and the next engine resumes. This repeats without restoring an abandoned C activation. A real exception follows the same LIFO order and enters the existing Mach/native disruption machinery.

The Mach VM reuses its existing cold exception edge as the cross-kind unwind token. A final disruption is prepared by a small cold helper before re-entering the VM’s ordinary resume dispatch. Both a direct edge and a second computed-goto edge from the resume block to the disruption label changed Clang’s layout of the giant VM function and caused double-digit pure-Mach regressions, despite being resume-only. Keeping the walk outside that CFG restored recursive-fib Mach from about 60.5 ms to 51.1 ms (pre-suspension composed baseline 49.7 ms). The helper mirrors the existing handler walk and uses the context’s permanent suspended-frame root while stack capture can trigger moving GC.

apply is a real call

The old QBE apply helper called Pit_CallInternal underneath a live helper C activation. A suspending native callee therefore could not be represented. apply now prepares an ordinary callee frame, stores it in the result slot, and signals the flat dispatcher just like frame/setarg/invoke. Non-functions retain the language’s identity result. Arguments and the prepared frame are rooted across allocation. This changes no mcode operation.

Poll policy

Calls and loops use different cheapest-correct mechanisms.

• Native Pit calls charge one shared reduction in their existing frame-prep or dynamic-call helper. The ordinary path does one non-atomic decrement and compare. The scheduler flag is read only when the 65,536-call production budget expires. An attempted relaxed atomic load on every call made fib about 14% slower and was deleted.
• Pure loops count down a QBE-local 1,024-iteration tranche. Only tranche exhaustion calls C and reads the scheduler flag.
• Production does not manufacture a yield merely because its accounting budget expires. It resets and continues unless the scheduler requested a pause. The focused test hook encodes a negative budget to force deterministic yields.
• A backedge poll is omitted only when exact CFG reachability proves that every path around the cycle crosses a charged call/tail_call. Removing those call nodes must make the latch unreachable. Closure’s call loop therefore has no redundant backedge poll.
• A zero-based, +1 counted loop with immutable constant bound <=64, no nested backedge, and no external entry is charged once at its exit. Mandelbrot’s inner 50-iteration loop qualifies. This removes a branch from each pixel iteration while retaining the exact reduction charge.
• Native callees reached through the flat apply/invoke dispatcher charge at the same boundary as inline calls. The dispatcher pushes the prepared callee first, then may release the C stack with that entry frame rooted at PC 0. This closes the call-only recursive-apply case; relying on loop charging had left such recursion unbounded.

The 1,024 pure-loop bound is a policy tradeoff: it lowers slow-path transitions fourfold from the 256 prototype while bounding scheduler-pause observation to 1,024 iterations. Calls observe within at most 65,536 call boundaries; on this machine that is well below one millisecond in call_hot.

I also prototyped omitting a call charge when a prior charged call dominated the site (the second recursive call in fib). It moved the native median only from about 32.307 to 32.106 ms, roughly 0.6%, while adding CFG analysis and a charge flag to three helper ABIs. That arm was deleted. The final emitter charges every Pit call boundary; there is no dormant alternate convention.

Correctness matrix

The focused C runner sets a tiny deterministic budget and optionally runs a copying GC between every resume. All final fixtures were built with the final emitter/runtime pair.

The ordinary production load of the apply fixture also returns 5002 without a manufactured suspension. Repeated suspensions consume/clear their resume stamps; moving collection rewrites every retained frame root.

The caught-disruption row is a composed-branch regression added after the first prototype. The native leaf disrupts after its Mach caller has already been frozen. The resumed Mach chain must enter the handler before executing the post-call addition; returning 778 proves the handler’s 777 result, rather than the invalid call result, is what the native outer caller receives.

Controlled performance

Eleven-sample medians, candidate and detached baseline measured sequentially on the same machine. Results match Mach in every row.

fannkuch samples have visible thermal/run-order drift, so treat the apparent win as “no detected regression,” not as an optimization claim. Likewise this work should not be sold as making executables faster. It makes native execution correctly schedulable for a median 0-4% tax on representative code, with one 12.3% recursion outlier.

A final candidate-only rerun after adding dispatcher call charging produced 7.065, 32.401, 32.165, 28.618, 208.116, and 175.603 ms respectively. Every result still matched Mach and every generated IL/assembly size matched the table below. The paired/interleaved table remains the comparison result; these candidate-only samples confirm that the final correctness fix did not open a new macro path.

The rejected scalar prototype was +11.7% call_hot, +9.9% closure, +13.8% fib, +15.6% mandelbrot, +0.3% fannkuch, and +0.8% spectralnorm. The continuation-stack version decisively beats it.

Mach has no new ordinary opcode-path work. Candidate/base observations were: call_hot 14.274/14.085, closure 59.243/56.067, fib about 50.86/49.22, mandelbrot 93.021/92.561, fannkuch 290.596/292.462, and spectralnorm 272.074/268.220 ms. These were not stable enough to claim a Mach regression or win; the only Mach change is on cross-engine suspension/disruption edges.

Composed-branch performance addendum

The final suspension design was replayed on the complete 7e3bbc90 compiler and Mach stack and then composed with A7’s native checked-load fold. This is a correctness feature, so the comparison is the tax relative to the already-fast branch, not a speed claim:

Recursive native fib remains the deliberate worst case: it has no loop backedge, so fairness must be observed at calls. The other shootouts show the intended low-single-digit tax. The first composed resume implementation made Mach fib about 60.5 ms solely through a cold CFG edge; that implementation was rejected and is not present.

Composed code size is also somewhat larger than the original isolated report. Fib grows from 59,801 to 62,986 QBE IL bytes (+5.3%) and 46,409 to 48,291 assembly bytes (+4.1%). Spectralnorm grows from 169,495 to 184,086 IL bytes (+8.6%) and 110,740 to 121,998 assembly bytes (+10.2%). This adds executable text, not heap/frame/per-actor metadata. The runtime-memory costs below remain unchanged.

Code-memory cost

The absolute mandelbrot increase is 7,077 assembly bytes, not the rejected prototype’s 34,622 bytes, but it is still material for GBA/N64-class targets. The next size pass should outline the function-local slow poll/resume dispatcher without adding a hot call or semantic opcode. Compile speed is not relevant; executable bytes are.

Runtime memory cost

Clang’s exact host layouts:

• baseline NativeRTState: 104 bytes;
• candidate NativeRTState: 168 bytes (+64 bytes per context that initializes native state);
• continuation entry: 32 bytes;
• lazy four-entry chunk including link: 136 bytes.

Manual nan32/ARM32 layout projection:

• baseline/candidate state: 64/128 bytes (+64);
• continuation entry: 20 bytes;
• lazy four-entry chunk: 84 bytes.

An actor that executes native code pays the state delta. An actor that never needs nested cross-engine state allocates no chunk. Suspended execution retains the frames that were already live but adds no bytes to each frame. The 136-byte lazy chunk is a reasonable correctness trade on desktop/N64; for a 288 KiB GBA profile, a later custom scanner can reduce the root overhead only if measured actor density requires it.

Verification and tooling

• CCACHE_DISABLE=1 make: passed.
• ./pit test tests/compile.cm: 91/91 on the final composed branch.
• focused suspension/GC matrix above: passed.
• cold full suite: 832/833, with only vm_suite timing out while it was still being realized; immediate isolated vm_suite: 1086/1086.
• full warm suite: 1918/1918, including after the cold Mach disruption helper.
• deterministic fuzz: 3,733/3,733 at seed 20260713 on the composed emitter; the subsequent helper-only change was rechecked by both caught and uncaught forced-suspension disruption fixtures.
• host NativeRTState/chunk layouts were dumped from Clang, not estimated.

aot_bench/perf_one.ce gained two experiment controls: PERF_NATIVE_ONLY=1 skips Mach execution for fixtures that intentionally disrupt, and PERF_DUMP_MCODE=1 prints optimized main mcode. Its native-only logger had a bug: concatenating text(null) disrupted after a successful native run. It now prints mach_result=null explicitly.

One workflow trap remains: Meson updates build/libpit_runtime.dylib, while focused ./pit commands load the worktree-root dylib. make, not merely meson compile, is required after helper changes. A dedicated sync target or a runner that loads the build-tree runtime would prevent silent emitter/runtime mismatches.

Follow-up priorities

1. Outline slow loop-poll/resume code to attack mandelbrot’s remaining 7 KiB.
2. Rework call accounting so the reduction counter can stay in the native call ABI or another cheap shared location; accept only a design that remains exactly freezeable. Recursive fib is the proof benchmark.
3. Interleave Mach baseline/candidate runs to rule out the spectralnorm timing fluctuation.
4. Only after those, consider replacing embedded continuation roots with a custom context scanner for the smallest memory profile.

Source: plans/archive/perf-2026-07/perf-native-suspension-unified.md