Debugging

Opening a running system up is four endowments, and the axis that separates them is what each costs the machine while it is present. Every one of them is a thing a build is compiled with, and a build compiled without one cannot grant it. A build that carries none of the four is the fastest machine.

They are ordinary endowments: a unit claims one, the shop grants or denies it, and a provider linked into the binary supplies it. A claim with no provider in the running binary is an error when the actor starts, so what a shipped artifact can see is a structural property of that artifact and can be checked rather than trusted. See Endowments.

Each of the four is scoped by an actor token. Applied to the actor holding it, it looks at that actor. Applied with another actor’s token, it reaches an actor that one owns — reach runs down the tree an actor created.

$inspect

$inspect reads. It lists the actors the runtime is running, and for one of them it gives the frames on its stack, the locals and closures those frames hold, its mailbox queue, and the bytecode of a function.

A frame is addressed by its depth, innermost first, and never by handing back a function value: a value out of another actor’s heap would be a dangling reference after that actor’s next collection and a capability besides. A slot comes back as its position, the role the frame ABI gives that position — this, arg, captured, local — the kind of value sitting there, and a rendering of it. The name of a local is not the runtime’s to give: no slot-name table exists anywhere in a running actor, because names live in the variable map beside the origin map. A debugger resolves {code identity, pc} against that map and names the slots itself.

Reading an actor’s frames and locals requires it to be not running, which for an idle actor is free. A busy one is frozen first — see $runtime below. That is the ordering the whole arrangement is built around: freeze, then read, then thaw.

An observer never blocks on a turn it does not own. Enumeration tries each actor’s turn lock, retries briefly, and past that reports the identity and state it can read safely rather than waiting — so listing actors is a read that always returns, and a mid-turn actor appears as running with its detail zeroed. A busy actor slows nothing down for the observer; it just answers less until it is frozen.

There is exactly one enumerator, and it sees everything the runtime knows about — which includes an actor whose shell exists but whose start has not finished. Such an actor is listed with the state starting: it has an id, an overling and a heap, but it has not entered the actor state machine and nothing has run on it yet. Anything less would leave an actor that exists unnameable, and the answer to that is never a second lookup beside the enumerator.

What comes back is a description: values are rendered, and any capability held in a frame appears as an opaque marker naming it. A description can be copied, sent, and printed without carrying authority with it, which is what lets a debugger run as an ordinary actor rather than as a piece of the runtime.

The one thing $inspect returns that is not a description is ref: given an id from its own enumeration, it hands back a live send target. That is deliberate and it is why the enumerator and ref belong to the same endowment — naming an actor you were not handed is exactly the authority $inspect is, and splitting the two would leave a shop able to enumerate for you and then hand you a target without you holding anything. A token cannot be manufactured any other way — see Security and Threat Model.

$inspect reads, and reading is all it does. It never writes a local, never rewrites a frame, and never changes what an actor is doing; a request that would change the debuggee is a different power with a different holder. Reading also costs the reader and nobody else — a build carrying $inspect executes exactly as fast as one carrying nothing, which makes it the endowment to grant when you want to see a system and leave its speed alone.

$hook

The VM raises a hook at every call and every return. That is the whole of what $hook is — a place to be told, at every call site and every return, that control passed through — and tracing is that hook reporting to a sink and letting the turn continue.

The hook check is what $hook costs, and it is paid at every call and every return. A build compiled without it has no hook to check, and runs at full speed.

What $hook installs is a fixed recorder into a bounded ring, not a callback of the holder’s. A hook fires between placing a callee’s arguments and entering it, with the interpreter holding raw frame pointers in C locals — running the holder’s code there would allocate, could collect, and would move frames the VM is about to dereference. It would also be arbitrary execution inside another actor, which is a write power wearing an observation’s name. So the debuggee appends events and never blocks; the debugger drains them on its own turn and decides there what a breakpoint means. The ring overwrites rather than stalls, and reports how many events it dropped: a hook that could stall the debuggee would change the program it is watching.

Suspending a turn

Suspending a turn where it stands and resuming it later with its frames intact is the scheduler’s, and it is a power of its own, separate from both readers above. It sits on the acting side of the line this page draws further down: $inspect observes and changes nothing, $hook observes control flow and changes nothing, and stopping an actor mid-turn changes what that actor is doing. So it is granted separately, and a build that wants to look at a running system without being able to halt it grants the readers and not this.

A breakpoint is the two halves together: the hook recognizes the site, the scheduler suspends, and the frames stay exactly as they were until something resumes them.

A suspended turn keeps its state in the actor rather than on a host stack, so this works the same on every target, and resuming continues the turn from where it stopped. A frozen actor is an ordinary actor that is currently idle.

Stepping

A step is one Mach word. The actor is thawed with a budget, runs exactly that many words, and parks itself again with its frames, pc and slots as one more word left them — the same parked state a freeze produces, so every reader above works unchanged on the new position. Stepping needs no hook: it is the freeze with a countdown, and it lives beside suspend_actor for the same reason freezing does.

It parks, and never disrupts. The two are different powers wearing similar words — an interrupt unwinds a turn and a park preserves it — and a debugger that could only interrupt could look at a program once.

A step requires the actor to be frozen already, and refuses otherwise. Stepping from wherever an actor happens to be has no defined start; the refusal is by name rather than an approximation.

Two boundaries the count does not cross, both forced by the mechanism rather than chosen:

• A nested VM entry is one word. Suspension saves a frame and a pc into the actor; it cannot unwind a host frame that called back into the VM. So a native callee that re-enters runs to completion and the step lands after it. A Pit-to-Pit call is not this — it stays in the one dispatch loop, so stepping walks into a Pit callee.
• A turn end freezes rather than releases. If the turn runs out of words before the budget does, the actor is frozen between turns instead of handed back to the scheduler, so a session survives a turn boundary. The reading that says which happened is vm_suspended: true means the step landed mid-turn and there are frames, false means the turn ended under it and the stack is empty.

The always-thaws rule. A debugger that leaves an actor frozen has done more damage than the bug it was chasing, so every exit — success, error, a target that died mid-walk — resumes first and reports second. Stopping wins over freezing in the runtime as well, so a debugger that dies cannot wedge a system permanently; that is a backstop, not an excuse.

Which turns are freezable

Every backward jump is a preemption point, and a fused one is no different. The compare- and-branch superinstructions carry the same poll as the plain jumps they replace, so a loop whose body is nothing but integer arithmetic — no allocation, no call, no record — parks like any other. This is worth saying out loud because a loop that will not stop invites the guess that the optimizer removed its check, and that guess has been wrong every time it has been made.

What actually decides it is whether the turn is a nested VM entry. Freezing parks the VM by saving a frame and a pc into the actor; it cannot unwind a host frame. So when the interpreter has been re-entered from C — the dispatch loop called out to a C function which called back into Pit — there is no way to park the inner loop and hand control back to the scheduler, and every poll site declines. A freeze request on such a turn is honoured at the next backward jump outside the nesting, which for a long-running inner loop means not soon.

The case where that bites is the start plan. An actor’s module bodies and its top-level body run synchronously, on the thread of whoever asked for the actor, through C shims that re-enter the VM — so all of a program’s start is a nested entry. A loop there cannot be frozen and cannot be stepped, and pit debug will report the actor as idle every time it looks, because the actor genuinely has no turn of its own yet: the work is happening inside its starter’s. That is the honest reading and not a bug in the report.

Stopping is not subject to any of this. An interrupt raises a disruption rather than saving state, and a disruption unwinds through nested entries by the ordinary exception path, so it is honoured at every backward jump at any depth. That difference is what the runaway protection is built on — see below.

What stepping costs when nobody is stepping: nothing. The dispatch loop already carried one relaxed atomic load per word, guarding the allocation-and-profile sampler. Stepping shares that word rather than adding a second, so a build with nobody stepping and nobody profiling executes the same instruction sequence it did before — one load, one unlikely branch, not taken. Everything the step check does sits behind that branch. This is the same shape as PIT_HOOK_ARMED: the check that is free is the check that folds into one already being paid.

Freezing and thawing

A frozen actor is resumed from outside. Receiving a message would take a turn, and the actor is suspended, so freeze and thaw are scheduler operations rather than messages.

A separate debugger actor does the resuming, scoped to the frozen actor’s token. The resume goes to the scheduler, which un-parks the actor from outside. This is what cross-actor reach is for: it is the path back for a self-suspended actor.

The debugger being an ordinary actor is what makes remote debugging fall out for free. The deep reach into the debuggee is local, between two actors on the same runtime; the conversation with whoever is driving is ordinary messaging, which is location transparent. Only descriptions cross the wire. A cartridge built for debugging carries a small stub actor holding $inspect and $hook and talks to a development machine — the same arrangement, one hop longer.

$vm

$vm answers questions about memory. It reports the runtime’s total and each actor’s own, walks an actor’s heap into the list of objects on it, and measures the footprint of a single value. The walk happens in-process and hands back a description of the object graph, so the result travels like any other data.

Allocation-site sampling is $vm as well, and it stays quiescent until a holder sets a nonzero sampling interval, so an actor nobody is measuring pays nothing for it.

The accounting that makes those totals true — the per-allocation and per-collection bookkeeping — is the standing price of $vm. A build compiled without it allocates and collects without keeping the books.

$pgo

A Mach pool can carry a table of counter sites, one row per call site, and an actor allocates its counters when it starts. $pgo drains them: it takes an actor’s live counts and hands them back as observations addressed by mcode function and site ID, which the pool’s origin map supplies. Recording them in mcode identity rather than the pool’s is what lets a profile outlive the pool it came from. Those observations become a profile object that a later finalization consumes for inlining, layout, and specialization — see Profile-guided optimization.

The counters are the cost, and they are paid by the pool and the actor together: a build compiled without $pgo finalizes pools with no counter table and starts actors with no counter array.

Hot reload

Replacing the code a running actor is executing is a capability of its own. It changes the actor rather than observing it, so it sits beside the four above rather than inside one. It is $reload, and it is two calls:

A reload builds a new Mach pool — an ordinary pool, through the ordinary path — and installs it into a running actor. Each function in the new pool is matched to the function it replaces by a stable identity, the {mcode unit hash, function ID} pair that survives lowering, so the match holds even though the two functions live in different pools at different addresses. The IDs a pool carries exist for exactly this: matching, never lookup.

Both halves of that pair are used, and each does a different job. A FUNCTIONS row carries its stable id as the text <source hash>#<mcode function ID>. The source hash is what makes the id unique across units — and it is also, by construction, different in the new pool, because a reload is an edit. So the unit half is fixed by the pairing the caller already made (this running module, that freshly pressed pool) and the function ID is what is matched inside it. Its limit is the honest one: inserting a function ahead of others renumbers them, and the arity and shape checks are what catch the result.

pit reload <program> is the tool over the pair. It realizes the program again — a reload is an ordinary realization, not a special lane — pairs each running unit against a freshly pressed pool, freezes, swaps, and thaws. It never restarts anything: a refused swap leaves the actor as it was and says why.

Freeze before you read. census takes the target’s mutex, and a turn holds its own mutex for its whole duration, so censusing an actor first would block out exactly the long call a reload most wants to catch. And suspend_actor sets debug_suspended immediately — that flag is the request landing, not the turn stopping. The condition to swap on is debug_suspended and either vm_suspended (parked mid-turn at a backward jump, frames intact) or an idle state. running and slow both mean a turn is still in flight.

The link plan sets the granularity. A pool is what gets swapped, so what you can reload without restarting is exactly what you put in its own pool — the same lever that decides optimization scope decides reload scope, and there is no second knob. See Compilation and Shipping.

A module’s export record is stone, and the module body does not run again. Those two facts are what make the swap small. There is no re-initialization and no second record: the reloader walks the export record the module already returned and, for each function named on it, repoints that function’s bytecode pointer at the new stone code. The record itself is never written to — only the function objects it names are mutated, which is exactly what a stone record permits.

It follows that a module can export a text, and that text cannot be hot reloaded. Only a function carries a pointer to repoint; every other member is the value the module already produced, frozen.

The export shape must match. You cannot add or remove a member on the exported record. A mismatch between the old record and the new module’s exports is a warning or an error from the reload machinery, named as such — never a silent partial swap.

Live frames are not migrated. A call already running finishes on its old code and its old frame layout; the repoint takes effect at that function’s next call. Nothing unwinds and nothing re-enters, so no effect is ever performed twice — the frame-layout problem simply does not arise. Two residuals follow from the same rule: the old pool stays pinned until the last frame using it unwinds, so nothing it holds is collected out from under a live frame; and a closure handed out by a frame still running old code keeps that old code until it dies.

That rule is not free, because the VM re-derives a frame’s instruction cursor from its function object on every resume and on every return into it. Repointing a function a live frame is executing would resume that frame at its old pc inside new code — the migration this rule forbids, arrived at by accident. So before anything is repointed, every live frame running one of the functions about to change is given a private copy of its function object, still bound to the old pool, and the frame is pointed at the copy. The frame then finishes on old code with nothing shared, and the record’s member is free to move. The copy keeps the old realization state, which is what pins the old placement for exactly as long as that frame lives — the first residual, made true by construction — and a closure the frame creates afterwards takes its cursor from the copy, which is the second. swap reports how many frames it pinned this way.

The export shape must match, and the swap says whether it checked. The names are taken from two places: the live record’s own keys, and the new unit’s exports declaration on pit.mcode.unit@3. A unit that declares no exports leaves the check unmade, and the reply’s shape_checked is false rather than the pass reading like a real one.

Granularity is the limit, and it is R7’s. A pool’s module-result operands are baked in the covering it was linked in, so a pool from one program’s realization is not interchangeable with the same source module’s pool from another’s. pit reload will only touch an actor that shares at least one pool with the realization it just built; --actor narrows it by hand. And a pool linked coarser than one unit per pool folds functions, so {unit, function ID} stops being 1:1 — the swap refuses it by name rather than picking one.

Every frame keeps its executable, unit-instance, pool, function, and position identity across a swap, so the origin map resolves equally against old and new code.

Debug information

Debug information is stored beside the artifact rather than inside it, addressed by the content hash of the code it describes. A build that does not want it does not carry it, and two builds of the same module share one copy.

The origin map is the sidecar that carries it. It is keyed by the pool’s content hash, and for each range of final instructions it names where that code came from:

frame.image_hash ──▶ origin map ──▶ { mcode hash, function ID, site ID,
                                      source hash, source span }

A stopped frame supplies that key itself. $inspect reports an image frame’s image_hash beside its function_index, so the pair a lookup needs comes off the frame with nothing inferred. This matters more than it looks: function indices are dense and pool-local, so a map for the wrong pool would not fail to resolve — it would resolve to a confident lie about a different program. With the key on the frame, picking the map is a comparison.

A frame also carries code_id, which is the mapping’s address. That answers “are these two frames in the same placement” and nothing else: an origin map is written long before any address exists, so code_id is not and cannot be a lookup key.

Everything on the right of that arrow is a content hash or a stable ID. No name mapping appears anywhere in the path — the map never says which locator a function was written under, because the pool does not know and does not need to. A folded range keeps its alternate origins in the same shape, and an inlined one keeps the whole call chain.

That is why one map serves debugging and profiling both. Turning a stopped frame into a source line is following it forwards; turning a counter into an mcode site is following it the same way. The frame’s own contents come from $inspect rather than from the map, so a stopped frame yields a position and its values together, from two sources that never have to agree about names because neither carries any.

Because the map is addressed by the pool hash, a running frame finds its own debug information by the hash of the code it is executing. Inspecting something already running is a lookup, and works wherever the artifacts are reachable.

Origin maps travel in a bundle’s debug slice, alongside the source trees they point at. A stripped product omits the slice and keeps the exact pool hash, which is all anyone needs to fetch the map from somewhere else later — so shipping without debug information never means losing the ability to add it back. See Packages and Distribution.

Symbols as files

A shipped program carries no Mach, so it cannot be symbolicated from itself. The pool’s content hash survives anyway — identity is not distribution — and that identity is already the map’s key, so making the map travel is two file operations rather than a new mechanism.

pit debug symbols export --out <dir> writes each origin map out as <pool-hash>.origin. The name is the bare hex with no blake2: prefix, because a colon is not a filename everywhere this runs and the prefix carries nothing the extension does not. The bytes are the stored bytes, unmodified — the same canonical encoding the shop holds — so an exported sidecar is byte-identical to the stored one, decodes with the same call, and can be checked against its own name. Which pools to export is named three ways because the hash arrives from three places: --actor takes the pools an actor’s frames are running, --cart takes every pool in a cartridge, and --hash takes one that arrived in a crash report.

A pool with no stored map is reported, never skipped. A half-exported symbol set that claims to be whole is worse than none: the gap surfaces later, on another machine, as a frame nobody can resolve.

pit debug --symbols <dir> is the other end. The directory is consulted before the shop store, because the case it exists for is a pool this shop did not build; the store answers as a fallback. The report says which route answered, since “resolved from the shop that built it” and “resolved from symbols someone handed me” are different claims about how much to trust the line. When neither answers, the gap names which remedy applies — a directory that does not carry the pool and a shop that never built it need different things done about them.

What a remote debugger would need

This is design, not code. The parts above already make a foreign pool resolvable locally; remote attach is the transport that has not been written.

Two conversations, and keeping them apart is the whole design. The deep reach into a debuggee — freeze, frames, locals, step — is local, between two actors on one runtime, because it touches raw frames and a heap that must not move. The conversation with whoever is driving is ordinary messaging between a stub actor and a pdb on a development machine, and only descriptions cross the wire. So a debug-flavoured cartridge carries the stub, the stub holds $inspect, $hook and $runtime, and nothing about the reach changes because a person is a network hop away.

What that leaves to specify is one thing: symbol fetch by hash. The debuggee reports each frame’s image_hash as it already does; pdb needs a map for a hash it may never have built. Three properties decide it, and all three are already true of the sidecar:

• The key is the pool’s content hash, so a request is a lookup and never a negotiation over versions, build ids, or paths.
• The bytes are self-checking — the name is the hash of the code the map describes, so a wrong or tampered sidecar is detected rather than believed. Symbols arriving from a third party is the normal case, not the exceptional one.
• The map contains no name mapping and no source text, only content hashes and stable IDs, so serving one reveals nothing about where the code came from. Source is a separate fetch against the source hash the map names, and it is legitimate for it to be refused.

That makes the remote piece a content-addressed read: a pdb collects the hashes off the frames it is shown, asks a symbol source for the ones it lacks, and resolves locally. The symbol source may be the shop that built the binary, a directory an operator exported and copied, or a bundle’s debug slice — the debugger cannot tell them apart and should not need to.

Log routing

Registering the process-wide log sink is $runtime, held by the actor boot selects and by nothing else. Its holder receives the runtime’s log stream and chooses which channels each actor emits.

It is an ordinary endowment: an actor holds it because its realization has the row, and the start path installs that row the way it installs every other. So who receives logs, like who may look into a running actor, is authority someone was given.

Which channels reach a subscriber is then the logger’s own decision, and pit log channel <name> [--off] is how you make it — see Boot. It silences the live stream, not the record: the observe log still has the line, so pit log and pit why still answer about a channel you turned off.

$runtime

The log sink is one of a family of powers that share a shape: they do not observe the machine, they act on it, and what they do is true for every actor at once.

Freezing lives here rather than in $inspect or $hook, and both halves of that are deliberate. It is not $inspect because $inspect is read-only, and stopping an actor is the largest change short of killing it — the worth of a read-only endowment is precisely that holding it cannot change anything. It is not $hook because $hook compiles out, and “stop an actor that is stuck in a loop” is a thing a build with no hook must still be able to do: nobody set a breakpoint, so no hook is involved. What is left is this table’s shape exactly — an act on the machine, reaching an actor you did not start.

A freeze gates scheduling, not delivery. Letters keep arriving and keep queueing; what stops is the actor being handed to a worker. It takes effect on a turn that is already running, because the request also asks the VM to park at its next backward jump — so a frozen actor is off every worker with its frames exactly as they were, which is what makes it safe to read. Asking is not the same as it having happened: suspend_actor returns whether the request reached a real actor, and $inspect.snapshot()’s debug_suspended is the reading that says it took.

Stopping wins over freezing. A death notice thaws its recipient, and halting an actor clears its freeze. Otherwise a frozen actor coupled to the root would never observe the root’s death, and a debugger session left open would hold pit down forever.

It is held chiefly by boot, or by an early actor wiring the machine up, and by nothing else afterwards. The matching reader, runtime_info(), is not part of it and needs no endowment: every actor reads node identity to stamp the tokens it creates. Reading what the node is costs nothing and reveals nothing; deciding what the node is, is authority.

$inspect observes, $runtime acts, and stopping is the pair that makes the line visible. Listing the actors a runtime is running is a read, so it is $inspect.snapshot() and there is exactly one enumerator. Reaching into that list and killing something is not, so it is here. $stop is the third case and stays where it is: stopping your own underling is ordinary parenting, not an act on the machine.

shutdown stops the root actor rather than raising a flag in C. Everything is transitively $coupled to the root, so its death cascades through the ordinary stop-watcher path and the engine exits when the last actor is removed — orderly, supervisable, and made of the same parts as every other actor death. See Booting.