Archive — history, not state. Kept for its reasoning and its evidence; its plan is closed.

Compiler optimization campaign: final performance matrix

Status: final integrated campaign seed measured and validated. External-runtime lanes are the earlier same-machine comparison run; Pit lanes below are the clean-daemon final branch measurements.

Date: 2026-07-14

Branch: compiler_optimizing

Committed base: da2a46b5901ac3f56ae9ba34b0550b832c90fa1f

Machine: Apple arm64 (Darwin 25.5.0, RELEASE_ARM64_T6050).

This matrix is the composed end-of-campaign measurement, not a collection of best historical numbers. It keeps dynamic/interpreted VM comparisons separate from native/JIT comparisons and verifies each workload result in the harness.

Interpretation groups are:

• Pit Mach, Lua, QuickJS, CRuby, and mruby are VM/interpreter comparisons.
• LuaJIT, Node/V8, Bun/JavaScriptCore, and .NET/RyuJIT are optimizing-JIT comparisons; they are the relevant aspirational field for hot AOT output, but they may use runtime feedback and allocation sinking unavailable to a cold static build.
• Pit native and ocamlopt are AOT comparisons. OCaml is compiled without a flambda flag in this matrix.

The table is deliberately about language workloads rather than identical object representations. JavaScript/Lua/Ruby use their normal language record objects. The statically typed ports use Dictionary/Hashtbl for the honest generic string-keyed record rows; a C# class or OCaml record would instead be a static offset-layout ceiling. Pit’s new dense shapes mean record_new now sits between those categories, while the polymorphic d_field and record_field sites remain the more useful generic-access comparisons.

Method

The comparison sources are aot_bench/cross/bench.*, aot_bench/cross/shootout.*, and aot_bench/cross/calls.{erl,exs}. Each micro/shootout program uses its existing calibrated harness: run the workload once for an oracle, then for each of three inner trials double repetitions until at least 50 ms and retain the best normalized time. This campaign executed each complete harness three times serially and reports the median of the three reported per-row values. Thus every table entry is a median of three independent calibrated best-of-three values. No benchmark programs ran in parallel.

Exact commands, each repeated three times:

lua aot_bench/cross/{bench,shootout}.lua
luajit aot_bench/cross/{bench,shootout}.lua
qjs aot_bench/cross/{bench,shootout}.js
node aot_bench/cross/{bench,shootout}.js
bun aot_bench/cross/{bench,shootout}.js
ruby aot_bench/cross/{bench,shootout}.rb
mruby aot_bench/cross/{bench,shootout}.rb
ocamlopt -O3 -o /private/tmp/pit_cross_{bench,shootout} aot_bench/cross/{bench,shootout}.ml
/private/tmp/pit_cross_{bench,shootout}
dotnet run --file aot_bench/cross/{bench,shootout}.cs -c Release
erlc -o /private/tmp aot_bench/cross/calls.erl
erl -pa /private/tmp -noshell -s calls main
elixir aot_bench/cross/calls.exs

Runtime versions: Lua 5.5.0; LuaJIT 2.1.1783773675; QuickJS 2026-06-04; Node 26.5.0 / V8 14.6.202.34; Bun 1.3.14; CRuby 2.6.10p210; mruby 4.0.0; OCaml 5.5.0 native (ocamlopt, no flambda flag); .NET SDK/runtime 10.0.301; Erlang/OTP 29 with BEAM JIT; Elixir 1.20.2 on OTP 29. s7 is not installed.

The macOS /usr/bin/time -l facility could not read kern.clockrate inside the benchmark sandbox and did not emit peak RSS. Cross-runtime peak memory is therefore unavailable in this run. Target heap/allocation/GC evidence for Pit comes from vs.ce’s actor statistics instead.

Cross-runtime micro suite

Milliseconds; lower is better. These are the fourteen workloads currently implemented consistently in every bench.* source.

mruby 4.0.0 rejects the 300,000-element array_grow workload with array size too big; this is a missing result, not a zero or a timeout.

Cross-runtime composed shootouts

All runtime results matched the shared workload oracles.

BEAM call comparison

These are seven-sample medians inside each run, then the median of three complete process runs. dynamic_call0 measures indirect selection among eight zero-argument immutable BEAM closures. It is useful call-dispatch evidence but is not semantically identical to Pit’s mutable captured-cell closure fixture.

Pit-only extended matrix

The Pit harness performs two warmups followed by seven timed samples per lane and reports the median. The final serialized 68-row aggregate completed with check=Y for every row. The selected rows below are the clean-daemon medians; allocation and collection columns are shown where the final summary captured them. Unlike the earlier exploratory matrix, nbody now statically binds its math/radians.sqrt dependency and executes in both lanes.

The nursery remained disabled throughout this final matrix: all nursery allocation and collection counters were zero.

Current readout

• Pure integer loops now land where intended: Pit native is essentially tied with OCaml/.NET on arith_int, arith_poly, and loop_nested; Pit Mach is competitive with or faster than Lua on those rows.
• Known/inlined calls are also healthy: call_hot native is close to OCaml and faster than LuaJIT. Honest dynamic activation is not: the polymorphic zero-argument call0_plain costs 40.65 ms and the captured/dynamic closure row is 3.2x OCaml and 2.3x LuaJIT. The retained Mach outer-frame cache does help the closure row, but it does not change native activation. The BEAM indirect-zero-arg result around 26 ms is a useful directional comparison, although BEAM’s fixture has immutable closures and does less accumulator arithmetic than Pit’s fixture.
• Boxed array traffic is a clear remaining native weakness. array_read is no faster native than Mach, 2.4x OCaml, and 1.7x LuaJIT. This agrees with the rejected exact boxed-array proof: check removal did not remove the dominant representation traffic.
• Dense shapes materially fixed record allocation density: record_new allocates 16.8 MB rather than the prior roughly 43.2 MB and is faster than the ordinary Lua/QuickJS/Ruby lanes. It is still allocation-bound and far behind JS JIT object allocation. Honest polymorphic record_field remains 2.4x LuaJIT but now beats OCaml’s Hashtbl analog.
• The shootouts identify different ceilings. Native mandelbrot is 4.5x faster than LuaJIT and within 2.2x of OCaml, so scalar numeric lowering is working. spectralnorm is 6.0x LuaJIT and 4.2x OCaml, dominated by calls and boxed array access. fannkuch is 18.5% faster than interpreted Lua, but remains 6.4x LuaJIT and 8.2x OCaml because of dense array/index mutation. binarytrees is slower than Lua and spends about 43 MB/27 collections; object allocation and collector behavior dominate it. nbody is an especially poor result (150.79 ms native versus 1.58 ms OCaml), showing that repeated array/record access and call boundaries still overwhelm otherwise healthy scalar arithmetic.

Movement from the contained campaign baseline

compiler-optimizing-handoff.md records the fresh contained-campaign baseline at 81685c37. It predates several mechanisms already present at this arc’s da2a46b5 starting point, so this table measures the full contained compiler journey rather than claiming every delta for the linked/shape/PGO arc alone.

The final same-artifact canary revises the earlier optimistic shape result. Dense/default binarytrees measured 74.79/47.25 ms and 43.25 MB, versus 83.63/44.86 ms and 53.96 MB for generic-template and 80.37/51.66 ms with shapes off; a generic repeat was 80.34/44.73 ms. Dense shapes are therefore a clear memory and Mach win and beat no-shape native, but they carry a repeatable approximately 5% native regression versus the generic-template representation on this one allocation-heavy workload. They remain retained because the shared representation cuts each common three-field record from 144 to 56 bytes, removes roughly 20% of binarytrees allocation traffic, and improves the default path on both lanes relative to no shapes. The native regression is an explicit follow-up target, not hidden as noise.

PGO production result and artifact identity

The final program-scoped profile run returned 3,200,000 in every mode:

The generation-1 portable profile is 7,088 bytes. Its semantic hash is d28969b9ced728b4e4753a5c998de137131819c715560fe0c05d50042db1f91f and its content hash is 48c924e48b69c3e487cac489e4fc988a4ec3a2dcd4a8db8ce4bde1e2409af9a9. Consumption uses only record-load sites with at least 1,024 observations and 90% valid top-four shape coverage.

Final strict artifacts contain 65 validated mcode modules. Exact sizes are 74,842,189 bytes for boot.qop, 67,509,488 bytes for boot/firmware, and 1,573,945 bytes for boot/boot (143,925,622 bytes combined). Compared with the immediate pre-audit campaign artifacts, the combined result is 1,095,242 bytes smaller; the engine is 126,372 bytes larger because it contains the strict portable validator/hydrator.

The final SHA-256 identities are a58afba087c65d9d04b31c10ade9c16400fc3766b3a3f8a8295749462eeb7d57 for boot.qop, 797db28fc9b1bec61db89da36a5656049db9da3389ce72a6f0dc61c12b175e89 for boot/firmware, and 6fff594fcb6f130c9c2ab5db5c1f595081320ec56d1c7f6f11ea1ebb4bf17174 for boot/boot.

Integrated validation and memory hygiene

• make seed passed; Meson passed 7/7; focused compiler tests passed 145/145.
• The isolated VM suite passed 1,086/1,086 and the warmed full suite passed 1,921/1,921. One cold full attempt timed out only while charging realization to vm_suite; its isolated and warmed reruns passed.
• Deterministic fuzz passed 3,733/3,733 checks over 500 programs with seed 20260713. The 68-row aggregate returned check=Y for every row.
• Final validation found and fixed two general wrong-code hazards: LICM no longer treats branch-local/conditional writes as dominating replacements, and QBE now records helper-written is_array, is_func, and is_record predicate destinations as frame writes.

A clean node mapped 64 MiB: 4 MiB of actor heap, 1 MiB of node constants, and about 20 MiB of shared Mach code; the shop was about 2 MiB live in an 8 MiB block and the builder about 59 KiB live in a 256 KiB block. Running the entire aggregate and test fleet without restarting grew the node to 704 MiB mapped, 278 MiB of actor heaps, 37 MiB of node constants, and a 242/256 MiB builder block. That is compiler/test-daemon retention, not target benchmark memory: it slowed binarytrees to 216/136 ms and record_new to 70/90 ms while leaving call_hot near 15.5/6.67 ms. Restarting restored 78/48 and 23/15 ms. Final performance claims use only the restarted clean-daemon measurements.

Source: plans/archive/perf-2026-07/perf-campaign-final-matrix.md