Working state — a note taken while the work happens, not a specification. The system as it is meant to be is in Architecture.

C3 — the send path, measured

Verdict: the two process-global send-path locks are NOT the next image_lock. C3 closes with evidence and no scheduler change.

The audit’s 6.1 predicted that actors_mutex + engine.lock were the remaining O(1)-per-message global serialization point, and named “give timer_heap its own mutex” as the cheapest first step. It said, in its own words, “I did not measure; this is a code-shape prediction, and it should be measured before it is fixed.”

It is now measured. The prediction is wrong about which lock, and the proposed first step is worth 0.4 % of engine-thread wall time. What the profile actually names is a per-actor lock — target->msg_mutex — at 63 %.

Date 2026-08-05 · branch arc/lock-work · base dev 8791b2c91 · darwin arm64, host-nan64-v1, 15 workers · machine shared with other lanes (see “contention” below for why the conclusion survives that).

1. The workload

benchmarks/msg_pairs.ce + benchmarks/fixtures/msg_pair_peer.ce.

N independent pairs of actors ping-pong a fixed number of rounds with each other. The parent starts 2N children, hands each driver its peer’s token, fires the start gun, and then does nothing until every pair reports — so the traffic being measured never passes through the parent, and the parent is not the serialization point the number is about. One round is exactly two letters (the ping and its reply), and every letter walks the whole path: scheduler_ref_actor under the process-global actors_mutex, then target->msg_mutex, then set_actor_state → enqueue_actor_priority under the process-global engine.lock. Numbers are reported per letter.

The A2 scheduler counters are read before and after the flood, so harness letters are visible next to flood letters rather than hidden inside the wall time. Every run below reported errors: 0 and enqueue == dequeue.

./pit benchmarks/msg_pairs.ce --pairs N --rounds 5000

2. Wall, paired: 15 workers vs PIT_WORKERS=1

Fresh daemon for each arm; rounds = 5000.

The serial cost of a letter is ~18 µs (1 worker, saturated). Fifteen workers buy 1.9×, not 15×: parallel efficiency ~13 %. Work is available while that happens — mean queue depth at dequeue rises to 4.7 at 32 pairs on 15 workers, so workers are not starved, they are blocked.

Counter cross-check at 32 pairs / 15 workers: enqueue 328,856, dequeue 328,856 (flood accounts for 320,000; the rest is harness + start/stop lifecycle), queue_depth_peak 33, wakeup 103 — at 32 chains the workers stay hot and almost never park on wake_cond, which is why __psynch_cvwait is not where the time is. At 2 pairs, wakeup 14,192: the fleet park/unparks constantly, and throughput per chain is at its best.

3. Where the wall time goes — sample, 8 s at 1 ms, during a 32-pair flood

sample <daemon pid> 8 1. 21 threads sampled; 5 are idle workq/dispatch threads with no engine frame and are excluded. 87,360 samples across the 16 engine threads (15 workers + timer). Each __psynch_* leaf is attributed to its nearest non-pthread caller, and each call site’s offset is resolved against the binary’s disassembly, so every row below names one specific lock rather than “a mutex”.

Offsets were resolved with objdump --disassemble-symbols; e.g. send_message+512 is the return address of the bl _sys_rmutex_lock at +508, whose argument is [x19, #0x2c98] = target->msg_mutex, and send_message+44 is the return of the bl _sys_mutex_lock at +40 on the actors_mutex global. That distinction is the whole finding, so it is not left to inference.

Rolled up:

4. The decision

The literal threshold in the C3 brief — “lock wait is a clear double-digit % of runnable samples” — is crossed, at 76 %. But the brief’s rule pairs that threshold with one named change: give timer_heap its own mutex so timer traffic stops riding engine.lock. The profile prices that change exactly:

• Timer-thread contention on engine.lock is 358 samples = 0.4 %.
• The other 6,231 engine.lock samples (7.1 %) are worker against worker — enqueue racing dequeue. Moving the timer heap to its own mutex removes none of them.

So the audit’s proposed fix has a measured ceiling of 0.4 % and costs a change to scheduler locking. Making it would be measuring and then ignoring the measurement, which is the exact failure mode C3 exists to prevent. Not implemented. C3 closes: the process-global send-path locks are 13.5 % combined and are not the bottleneck at current scale.

This is a refusal of the named fix, not of the finding. If John wants the timer_heap split anyway on design grounds — it is mechanically right that timer traffic should not ride the run-queue lock — the number above is what it will buy, and it should be landed as a hygiene item with no performance claim attached.

5. What the profile did name — the real candidate, out of C3’s scope

target->msg_mutex at 62.9 %. Every critical section under it is short (send_message does an arrput; actor_turn pops one letter; actor_gc_scan walks the letters/parks/signals lists). Two facts make short sections expensive here:

1. It is a sys_rmutex — PTHREAD_MUTEX_RECURSIVE (platform/posix-runtime/source/sys_thread_pthread.c:58-65), which on Darwin selects the firstfit policy visible throughout the trace as _pthread_mutex_firstfit_lock_slow. Every collision is a full kernel park/wake, an order of magnitude more than the section it protects.
2. A worker parked here is out of the scheduler. It cannot pick up any of the other 31 runnable chains, which is what turns a per-actor lock into a fleet-wide throughput cap and why depth-at-dequeue climbs to 4.7 while efficiency sits at 13 %.

The 3.1 % under system_log_payload+696 is the same lock on the logger’s single mailbox — every logging actor in the fleet serializing on one actor’s mutex. Same shape, separate and smaller.

None of this is a “send-path lock split” and none of it is cheap: msg_mutex is recursive on purpose (scheduler.c:1174-1181 — pit_park_create re-enters it), it is the lock that serializes park delivery against actor_gc_scan, and its correctness argument (“a pending park is observed in exactly one of parks_head or actor->letters at every observable moment”) is load-bearing. Making it cheaper is a real piece of design work — a lock-free or two-lock mailbox with the park invariant preserved — and belongs in its own lane with its own rulings. It is recorded here so the next reader starts from the measurement instead of from the code shape.

6. Contention caveat

Another lane was building throughout. That inflates absolute wall figures but does not touch the conclusion, which rests on the ratio between call sites inside one daemon’s own threads — a competing process steals CPU from all sites equally. The paired 1-worker arm is the control: it removes contention entirely and shows the serial per-letter cost, and the two arms were run back to back on the same machine state.

7. Reproducing

make
./pit down && ./pit ps                        # fresh daemon, 15 workers
./pit benchmarks/msg_pairs.ce --pairs 32 --rounds 5000
sample $(cat .pit/pit.pid) 8 1 -f /tmp/flood32.txt   # during a longer flood
./pit down && PIT_WORKERS=1 ./pit ps          # the paired arm
./pit benchmarks/msg_pairs.ce --pairs 32 --rounds 5000

Source: plans/proposal-notes/send-path-measurement.md