Archive — history, not state. Kept for its reasoning and its evidence; its plan is closed.

Native suspension experiment

Status: research-only; do not merge.

Superseded by perf-native-suspension-unified.md. This file records the rejected scalar-state prototype and its measurements; do not use its boundary claims as the state of the current experiment.

Baseline: 1ea3d92a.

This experiment asks the narrow question left open by the AOT call work: can native Pit code stop at a fair scheduling boundary, discard the host C stack, survive a moving collection, and later continue without abandoning the current fast native frame representation? The answer is yes for an all-native frame chain and for a Mach caller waiting on native success, but not yet for an arbitrarily nested Mach/native chain. The scalar suspended-Mach state in PitContext is the architectural blocker.

The implementation is deliberately not proposed for integration. It proves the vertical slice, identifies its steady-state and code-memory costs, and specifies the continuation stack needed by a follow-up.

What the prototype does

• The native lowering assigns target-internal resume segments to backward branches and call sites. Canonical mcode is unchanged and there is no new semantic opcode.
• A taken backedge decrements a QBE-local tranche counter. Once per 256 taken backedges it charges a per-context reduction budget (default 65,536).
• Native calls charge one reduction while already inside their frame-preparation helper. This matters because recursive computation may have no backedge.
• On exhaustion or pause, generated code spills register-promoted locals to its GC-walkable Pit frame, writes the resume segment to frame->address, and returns PIT_SUSPENDED up the host stack.
• Direct native callers retain the callee frame, stamp their own continuation, and propagate the sentinel. The scheduler requeues cooperative exhaustion without counting a slow strike.
• Resume starts from the retained top native frame and returns through the native chain stack-flatly. Consumed frame-address stamps are cleared before the continuation executes, so the same activation can suspend again later.
• A Mach caller can retain its frame while native code suspends, receive the eventual successful result, and continue through the existing scheduler path.

No Pit frame field or semantic slot was added. NativeRTState grew by 40 bytes on the host nan64 ABI. Suspended calls retain the frames that were already live; the prototype adds no per-frame bytes. Its significant memory cost is generated code, not actor heap.

Correctness bugs found while building it

1. A deep native callee unwinds through every C caller. Stamping only the top frame’s immediate caller loses outer continuations. Propagation must walk outward and stamp the first unstamped native caller.
2. A resumed frame must clear the continuation stamp it consumed. Otherwise a later suspension sees the caller as already stamped and skips it.
3. Nested dispatcher-created native frames were present in the AOT GC-root array but their Pit caller field was null. Normal execution hid this; spectralnorm suspension crashed while trying to spill the unreachable outer caller. Explicitly linking the frame fixed repeated suspension, including GC after every yield.
4. Spilling promoted locals before every call was correct but expensive. Spilling after the callee returns the suspension sentinel is safe only after re-deriving the exact caller from the GC-updated frame chain. Moving this work to the cold edge recovered roughly half of the initial call-path regression.
5. Returning a tagged prepared-frame pointer on the rare “defer entry” case added normal-path masking and branching. Returning the existing null/cold result and querying suspension only on that branch is cheaper.

Proven vertical slice

The focused runner loads a compiled dylib into a raw context, applies a small budget, optionally forces GC between every resume, and checks result, suspension count, and retained native depth.

This proves repeated loop suspension, direct recursive suspension, nested dispatcher native suspension, moving-GC pointer updates, stack overflow after prior resumes, and native disruption after prior resumes.

Boundary that is not solved

Native to Mach to native nesting is not correct with the current context shape. There is one ctx->suspended_frame_ref, suspended_pc, and frame mark. In this sequence:

1. outer Mach calls native;
2. native calls inner Mach;
3. inner Mach suspends and writes the scalar state;
4. native unwinds;
5. outer Mach then suspends and overwrites the same scalar state.

The inner continuation is gone. No ordering trick can recover it after the overwrite. A similar boundary exists when native eventually disrupts while a suspended Mach caller is waiting. A direct check in the Mach resume entry makes that case correct, but A/B measurement showed that the added edge perturbs Clang’s optimization of the enormous register-VM function: arith_int Mach moved from about 20.7 ms to 28.7 ms and compiler realization began missing the VM-suite deadline. That arm was deleted. Successful Mach-to-native suspension is retained; disruption delivered through that bridge is not claimed.

C modules remain a suspension barrier, as required: native code does not try to serialize an arbitrary C activation.

Controlled performance

Both sides were built in detached worktrees from the same 1ea3d92a baseline. Numbers are 11-sample medians from aot_bench/perf_one.ce. They measure the cost of the current fairness mechanism, not an executable-speed win.

The mandatory part of the tax is a bounded poll: calls must observe pause and a long loop must check often enough that an actor cannot monopolize its worker. The current exact implementation is not mandatory. In particular, all of these are prototype choices and remain optimization work:

• decrement/compare/branch at every taken backedge;
• a context reduction update on every native call;
• per-site resume blocks in the same QBE function, increasing register pressure even on the normal path;
• a linear resume-segment dispatcher;
• separate native and Mach suspended-state mechanisms.

Call optimization and fairness must be evaluated separately. The baseline fast frame prep/pop remains useful; fairness added about 14% to recursive fib here. The initial eager-spill arm was about 25% slower, so cold-edge serialization is already a material improvement, but this is not a satisfactory final call ABI.

Code-memory tradeoff

IL and assembly byte counts come from the same focused builds.

The mandelbrot result alone rejects this resume-block layout for small-memory targets. The common 5-10% growth is also material for N64/GBA-class budgets. No amount of compile-speed indifference makes a 2x executable acceptable.

Smallest correct unified continuation stack

The follow-up should replace the scalar Mach suspension fields and the bridge booleans with one per-context LIFO of execution states. A compact custom-scanned entry is:

typedef struct {
  PitValue frame;       /* GC-updated owner/leaf frame */
  size_t frame_mark;    /* Mach frame-stack restoration mark; zero for native */
  uint32_t resume;      /* Mach PC or native target-internal segment */
  uint16_t dest;        /* result slot, 0xffff for none */
  uint8_t kind;         /* MACH or NATIVE */
  uint8_t flags;        /* LEAF/AWAITING, TAIL, DISRUPTED */
} PitContinuation;

This is 24 bytes on nan64/64-bit hosts and 16 bytes in a nan32/32-bit profile. Embedding a PitGCRef instead of using one context scanner makes it 32 bytes on the host, so the custom scan is worth having. Allocate stable chunks of four entries lazily: roughly 112 bytes including a host chunk header, or about 72 bytes on a 32-bit target. An actor that never crosses or suspends allocates no chunk.

Ownership and invariants:

• The owning PitContext and its scheduler worker are the only mutators.
• Push the caller’s AWAITING entry before crossing Mach/native kinds. This establishes the right order before a child can suspend.
• A yielding leaf pushes/updates the top LEAF entry. After it completes, pop it, deliver its value to the preceding AWAITING entry, then resume that entry. Repeat stack-flatly until a new yield or the outer result.
• Mach-to-Mach and native-to-native calls keep their existing intra-VM frame chains; entries are required at cross-kind boundaries and for the suspended leaf, not for every language call.
• GC scans every active entry’s frame and overwrites it with the forwarded value. Inactive chunk cells must be null. Native frames remain in the existing AOT root stack during the pilot; duplicate roots are safe and can be removed only after a separate audit.
• The stack bound is ctx->stack_limit with a hard representational ceiling of 65,535. A push that would exceed it raises stack overflow before entering the child. Allocation failure raises OOM before crossing. Never drop an entry, overwrite the oldest state, or fall back to an unfreezable Pit call.
• Pop only after the result or disruption has been installed in its owner. Disruption uses the same LIFO delivery, so handlers run in the correct VM.
• C calls are explicit barriers and do not create serializable entries.

This stack is prerequisite infrastructure, not an executable optimization. Once it is correct, optimize the hot poll independently: measure time-based pause-only calls versus cooperative call tranches, hoist loop polls to selected latches, and split cold resume code so it does not extend normal-path liveness. Keep the requirements: bounded pause latency, no host-stack retention, and no new per-frame memory.

Verification and tooling notes

• CCACHE_DISABLE=1 make: passed. The sandbox cannot write the host ccache directory; plain make fails there even though the project is healthy.
• ./pit test tests/compile.cm: 73/73 passed.
• Focused native suspension fixtures above: passed, including forced GC.
• Full sandboxed suite: 815 passed / 7 failed; six are denied bind/socket operations and one is the known cold VM-suite realization timeout.
• Full unrestricted suite: 832 passed / 1 failed (vm_suite realization timeout). The isolated run also hit the deadline while rebuilding. No test assertion failed, but this is not a green full-suite result and is another reason not to integrate the branch.
• meson compile updates build/libpit_runtime.dylib but focused ./pit commands load the worktree-root dylib. Running make is required after helper edits; otherwise the generated dylib and runtime silently disagree. A dedicated make native-dev-sync target or having the focused runner load the build-tree dylib would remove this trap.

Disposition

Keep this commit as a research artifact only. It proves that Pit native frames can be frozen and resumed without changing semantic mcode or adding per-frame memory, and it found the caller-link invariants needed for GC. It does not meet the language’s complete cross-kind call contract, its hot tax is 0-16% on the measured corpus, and its code growth reaches 102%. The next implementation experiment should start with the unified continuation stack, then reapply only the smallest leaf/backedge slice and use the performance/code-size gates above.

Source: plans/archive/perf-2026-07/perf-native-suspension-experiment.md