Build and Artifacts

The pipeline

Compilation has two halves that answer different questions. The first is portable and per-unit; the second is target-final and works over groups.

source ──▶ AST ──▶ mcode                    (one unit at a time, portable)

one or more mcode units ──▶ Mach pool       (target-final, linked and optimized)
one or more mcode units ──▶ native image

The requestor graph

$start(locator) is the public requestor that opens the graph. It asks for the locator’s executable, which asks for the closure, which fans its members out with parallel. Each member asks independently for source bytes and cached mcode or Mach. The diamonds below are decisions; the work boxes are requestors.

Builder infrastructure has a separate generation-boundary input. The outgoing generation compiles the incoming coordinator. The incoming coordinator then realizes its builder executable once, records that manifest hash as its compiler/lowerer identity, and pins the executable. A unit cache miss starts a new actor by supplying that pinned executable to $start. Supplying an executable means instantiate: it cannot re-enter locator resolution, freshness, or realization.

flowchart TD
  subgraph generation["Generation boundary — once"]
    G["generation G coordinator"] --> NG["realize generation G+1 coordinator"]
    NG --> H["handoff(manifest hash)"]
    H --> RB["realize builder executable once"]
    RB --> PIN["pin builder executable + manifest identity"]
  end

  START["$start(locator)"] --> RF{"realization cache hit?"}
  RF -->|yes| SPAWN["instantiate requested executable"]
  RF -->|no| SF{"single-flight(realization key)"}
  SF -->|join| WAIT["wait for the one producer"]
  SF -->|claim| E["executable(locator, plan)"]
  WAIT --> SPAWN

  E --> C["closure(locator)"]
  C --> W["parallel: walk import frontiers"]
  W --> F["parallel: all closure derivations"]
  F --> U["unit_mcode(locator)"]
  F --> O["object(c_file)"]
  O --> D["dylib(package)"]

  U --> SRC["file_contents(locator)"]
  SRC --> MK["mcode_key(source hash, compiler manifest, options)"]
  MK --> KC{"mcode cache hit?"}
  KC -->|yes| M["mcode content"]
  KC -->|no| S1["$start(executable = pinned builder)"]
  PIN --> S1
  S1 --> CP["compile one unit"]
  CP --> MS["store mcode + derivation"]
  MS --> M

  M --> B["unit_blob(locator, target)"]
  B --> BK["mach_key(mcode hash, lowerer manifest, target)"]
  BK --> KB{"mach cache hit?"}
  KB -->|yes| MB["mach blob"]
  KB -->|no| S2["$start(executable = pinned builder)"]
  PIN --> S2
  S2 --> LO["lower one unit"]
  LO --> BS["store mach derivation"]
  BS --> MB

  MB --> P["pool: link fragments once"]
  D --> X["store executable manifest"]
  P --> X
  X --> DONE["store realization + release single-flight waiters"]
  DONE --> SPAWN

unit_mcode, unit_blob, object, dylib, pool, closure, file contents, and the final executable are also public atoms; callers do not need to invoke the whole graph to ask for one result. Compiler and lowerer executable-manifest hashes are explicit derivation inputs. There is no global compiler salt, invalidation event, batch protocol, synchronous builder queue, or per-operation builder realization. Parallel width is an optional backpressure setting; its absence means unbounded fan-out.

The second arrow is the interesting one. Turning mcode into a pool is a lowering and linking step: it lowers each unit to the target’s final instructions, links the set together, optimizes across them, and lays out the result for one exact target. So a pool is made from a group of units, and how many units share a pool is a choice the build makes.

Pit emits, C reads

That step is a pit tool called a press, and inside it the work splits in two: the lowering decides meaning, the link decides placement.

The lowering takes one unit and a complete target profile and produces a pit.mach.fragment@1 — the instruction words in the profile’s byte order, a number’s exact bit pattern, a text object’s header and hash and packing in the profile’s text representation, and the record shapes. Every target fact it writes it reads from the profile record; none is compiled in. That is what makes pressing a cross operation: the same press produces a pool for this machine, for a console, or for the next Mach ABI, by being handed a different profile.

The pool itself is written in pit. The linker takes N fragments, dedups their stone by byte equality, interns finalized VALUES pool-wide, relocates and widens constant operands, rebases branches and PC metadata, patches pool-global shape operands, finalizes the import edges the covering decided, and lays out the sections. It never learns an opcode and never builds an object header — those are the lowering’s, and the two never learn each other’s job.

C writes no artifact. A fragment is an in-process handoff with no content hash that never leaves the press. C’s durable side is the reader: the validator that checks a pool’s structure at open, and the VM that executes it. C reads only what must be readable before any pit runs — the fixed header and the cart’s nota metadata — and every format C reads has exactly one writer, in pit.

The reader is the authority. A pool is correct because the target’s own runtime validates and runs it, and that is enforced mechanically: every profile arm the build system knows is exercised by pressing a fixture pool with the emitter for that arm’s profile and booting it under that arm’s C. A profile whose fixture boots is a profile the emitter speaks correctly.

The emitter also records, beside each pool, which mcode instruction produced each final word — which is what the origin map is made of.

Finalizing for a target

Any machine with a working pit system makes code for any machine; some machines have no compiler at all, and receive finished carts. That is ordinary cross-compilation, and it is one mechanism with no special cases:

  • A console cart is pressed on a desktop: the emitter is handed the console’s profile, the pools come out in that profile’s layout, and the cart is composed around them. The console maps and runs; it never lowers, links, or compiles.
  • A new Mach ABI is one more profile. When the ABI or a stamp lever moves, forge runs the predecessor’s Pit lowering for the new profile before it rebuilds C, so the new binary lands beside a cart it can mount. Every commit holds a source tree and cart that match. The mcode snapshot is ABI-neutral input, but is not a cold-boot escape: the engine which reads it is itself a generation-locked pool. See The Pipeline.
  • The compiler is an actor, and actors are the unit of shipping. A build that carries the compiler fleet can press carts; a build that does not, cannot — and that is the point. Compilation is something a running pit system does, not something a runtime binary contains, so leaving the compiler out of a shipped cart removes the capability entirely rather than disabling it.
  • Pressing is a build capability. Encoding a value for a foreign representation is the one target fact pit cannot spell for itself — each representation’s C header is the single writer of its own bit layout, and mirroring those layouts anywhere else is the system’s worst desync hazard. So a build whose recipe grants the press capability compiles each representation’s small ABI-only encoding unit alongside its own; a build without it — a console, a sealed player — carries none of them. One writer per encoding, present exactly where pressing is.

A target is a recipe, and pressing for one is a read of that file: the seven stamp axes come out of recipes/<target>.json through the single renderer that also produces the target’s build defines, so a cart’s header and the runtime that will mount it are spelled from one reading. pit cake profile <target> prints that string. Explicit levers (--numrep, --endian, --gc, --text, --profile-name) override it, because a build is allowed to diverge from every committed recipe.

The press consumes a module’s portable mcode unit, never a payload already lowered for some other machine. That is what keeps cross-pressing ordinary rather than special: the presser holds nothing target-final that it would have to translate. A realization the running system will execute is a different question with a different answer — that lane presses for the running profile and refuses to answer for any other, because handing back host bytes under a foreign stamp is the one outcome nobody can debug.

The acceptance test is make cross-check. On one machine it presses boot pairs for targets that machine is not, and proves each is stamped for its target, is structurally readable cross-target, and is refused by name by a runtime it is not for; CI runs the suites on the machines that are those targets. A system where that is green is a system whose emitter is parameterized by profile rather than by presser.

Products

A product is a cart with an entrypoint, and nothing more — no manifest file, no descriptor, no new artifact kind. It is produced by one command:

pit product <locator> [--include <locator>,...] [--target <system>] [--out <path>]

which is a formalization of two steps that already existed rather than a third one: pit cement --product presses the cart, pit forge --boot embeds it in a binary. What the command adds is closure discovery and the start declaration, so that a Makefile line is the whole definition of what is being shipped.

Closure discovery is static analysis of $start sites, and it is the same source closure walk the shop already runs. The compiler records every $start call site on the unit’s reachability record; a site whose second argument is a text literal carries that literal as its target. The closure walk resolves those targets against the same resolver that resolves a use(), records the edge on the importing file’s row, and enqueues the resolved locator — so a started actor’s imports and its own $start sites are walked in turn, and what comes back is the transitive set of actors this program can create. A site whose second argument is not a literal is recorded separately, with its file and source position, and reported to the caller; --include is the answer to it. Nothing is dropped silently in either direction, because a build that omitted an actor would produce a binary that dies at the moment it tries to start it.

Three artifacts, one command: a target realization per executable in the closure (the ordinary realize lane), one pit.cart@2 image pressed from the whole set, and one binary with that image linked into it as a boot section. Only the last survives by default; --keep-cart keeps the middle one for inspection with pit cart info.

Two things distinguish a product’s press from the development one:

  • The start declaration. One-shot, no shop, no daemon (see Boot), where the dev cart declares the zero word — resident, shop, daemon. That declaration is the only reason a shipped binary behaves differently, and it is in the artifact rather than in the environment.
  • Stripping. Producer names and diagnostic log channels are removed from the linked program before lowering; console, error, panic and disrupt survive, because a shipped program still has to be able to say what went wrong. Every other press — including the boot press that makes boot/root.cart — leaves both in place, so the development artifact keeps the producer information a stripped one gives up.

The cart root is chosen by what discovery found. An entrypoint that starts nobody is pressed as the cart’s own root, with no shop actor in the image; a program that starts other actors keeps the root shop beside it, because a start has to be served from the image. That is the only lever over R3a’s graceful degradation: the shop is asked for X, looks in its cart, and refuses by name when neither X nor a fetcher nor a compiler is there.

Portable mcode

An mcode unit is the canonical compiler IR for one source file. It carries the compiler-language and format stamps, the source content hash and unit kind, exact semantic literals, portable instructions and functions, raw import requests and static claims, stable function and site IDs, source spans keyed by site ID, and any durable target-neutral optimization facts the selected options produced.

Mcode stays deliberately free of target facts. The canonical package locator, checkout path, target profile, numeric representation, endian, object layout, final opcode selection, link grouping, and profile observations all belong to later layers. An mcode unit records the imports it requests; the executable manifest records what those requests resolved to.

Compilation supports optional target-neutral stages and flags through the compiler profile (distinct from the hardware/ABI target profile described above) and, beside it, the per-pass switches.

A profile is your selection of flags. There is exactly one named pass profile — ship, all passes on at both the unit and link stages, except join_types, which the link plan turns on for a whole-program plan through its own switch. Everything a second name used to buy is expressed directly: {profile: "ship", passes: {inline: false}} is what a size-oriented build asks for, and any individual pass can be turned off at either stage or at both. Two earlier names, dev and small, were deleted: dev’s link-stage set was byte-identical to ship’s, small differed by one switch, and dev’s unit-stage set measured worse on both axes at once — output within 0.32% of ship, and slower to compile.

The size programme is a separate lever from the passes. The stripping common to every emitted Mach pool (function display-name values omitted) is a producer flag on the link plan; general outlining and selective name/log stripping are not yet built.

ship is also what realization selects. Realization always yields a long-lived artifact — a resident that is started, compared against the seeded one by source closure, and kept in the store — so residents, boot services, press, publish, seed and bundle alike take it, and direct low-level shop_build calls default to it. An explicit compiler_profile realization input (the --compiler-profile flag) still names it for tests and measurements.

A resident’s identity folds every unit’s mcode hash, and the profile and switch selection move those hashes, so a resident realized under one selection can never match a seed pressed under another. Defaulting realization to a non-shipped selection therefore made every seeded resident stale on every start — that is why realization’s default is not a convenience choice.

These choices are semantic inputs. The compiler profile and the explicit disable list are both folded into K_mcode and the realization context, so a unit or realization cached under one selection can never satisfy another; when optimized bytes differ they also have distinct mcode content hashes.

For a source content hash S:

K_mcode = hash(S, compiler/language semantics, mcode format,
               compiler profile, selected target-neutral stages and flags)
K_mcode ──▶ mcode content hash M

Byte-identical source under identical compiler inputs reuses one source object and one mcode object, even when several locators name it. The rows that referred to it keep their own package, locator, version, binding, and policy provenance.

Executable manifest

The builder resolves one root .ce and its transitive .cm requests into an executable manifest: the root’s locator and mcode hash, one row per canonical package-file binding, the selected package version and tree, source and mcode content hashes, every resolved import edge, a deterministic module initialization order, root entry semantics, per-unit and aggregate claims, package lock identity, and the inputs policy will be asked about.

One selection per package, per executable. A package appears in the manifest under exactly one identity arm — one version, one tree — and <package>::<file> therefore occurs once in a closure. Aliases are the version pins, the lock records what they resolved to, and the manifest is where that resolution becomes a fact the executable carries. A closure that would need two versions of one package is a build error naming both paths, never a build that quietly picks. See Packages and Distribution.

The manifest is also the border table between names and hashes. Locators live here and at catalogs, because that is where humans and resolution meet. Below it, artifacts are named by content hash, and inside an artifact there are only dense indices and self-relative offsets — a pool does not know which locator a function came from, and does not need to.

Its content hash is the exact logical identity of the executable. Link granularity, placement, target profile, policy context, and final bytecode live in separate artifacts — which is what allows several different realizations of one logical executable.

Mach finalization

A link group is one or more executable unit instances finalized together. The pool emitter consumes the ordered mcode hashes and bindings, a link plan and optimization flags, the complete target profile, the Mach ABI and finalizer stamps, and optionally the hashes of profile observations.

It emits a Mach pool: a versioned header and complete profile stamp, entry and function descriptor tables, final dispatchable bytecode, immutable stone text and blobs, constants and constant-reference tables, immutable record shapes and key tables, nested-function relations, precomputed lookup structures, and checked relative offsets joining every section.

A pool is mapped and read in place. Its sections are already the target’s exact layout, so the accessors read STONE, SHAPES, and the instruction words where they sit. No per-load or per-pool materialized constant or object tree is built on the way in, and there is no ingestion pass that rewrites the bytes — mapping is the whole of loading, which is what makes execute-in-place from ROM the ordinary case rather than a special one.

References inside one pool are finalized as checked offsets or dense indices, so they cost nothing at run time. References between pools stay declared imports, and those are the only references a runtime binding resolves.

The link plan chooses granularity, and the choice is a real trade:

  • one mcode unit per pool — fastest compiles, finest-grained reload;
  • one executable closure per pool — optimization across the whole program;
  • several compatible closures per pool — optimization across a whole cart;
  • any deterministic partition in between.

Each pool carries fixed header, section-table, alignment, verification, and binding overhead, so finalization measures that overhead when it picks the granularity. The link plan decides optimization, constant pooling, identical-code folding, rebuild scope — and, because a pool is the unit of replacement, hot-reload granularity too. The placement plan — a separate choice — decides addressability and lifetime.

A pool carries the stable mcode function IDs its units were compiled with. Those are for matching — pairing a new function to the one it replaces across a reload, or an observation to the site that produced it — and never for lookup: dispatch inside a pool goes through indices, and nothing resolves a name at run time.

Native finalization

Native finalization takes the same ordered mcode and bindings plus a native link plan, the complete target profile and native image ABI, and the full content stamp of the native toolchain, SDK, linker, and runtime ABI. A different compiler, SDK, linker, ABI, flag set, target, link plan, or consumed profile object is a different derivation with a different key.

A native image runs with C-floor authority. It becomes eligible when the target realization selects the native execution form and device policy authorizes that exact image, its toolchain and signing authority, its claims, and its target. Its content hash verifies the bytes; trust is a separate question, answered by policy.

Target realization

A target realization is the startable artifact. It joins one executable manifest to one way of running it:

  • the executable-manifest hash;
  • the complete target profile;
  • the execution form, mach or native;
  • the ordered exact pool or native-image hashes;
  • unit-instance, module, function, and root entry bindings;
  • the start plan and module initialization rows, including each unit’s result slot and the name-to-value rows its bindings install;
  • the required provider and C-floor identities.

It is explicit on purpose. A runner reads rows and follows them — it does not compare the realization against the manifest, re-check claims, or re-derive an order. Anything a runner would otherwise have to work out belongs in the realization instead.

A canonical actor locator selects a compatible realization through a catalog — the last place a name appears on the way to running code — and actor startup follows the realization’s explicit pool and entry rows.

The realization’s binding rows are also the authority for what code can reach what. Two pools sharing a mapping, a cart, or a process are merely near each other; reachability comes from the binding rows alone. Confined Mach code receives exactly the pool references resolved during setup, and pool references are made by that resolution rather than derived from ordinary data. Offset and bounds checks protect structural integrity; the realization grants authority.

Two hashes, and freshness

A content hash names bytes — hash(canonical artifact bytes). It is how an artifact is stored, fetched, deduplicated, verified, and signed.

A derivation key names a computation — hash(every semantic input to one transformation). Looking it up yields an output content hash.

content store:      content hash   ──▶ canonical bytes
derivation catalog: derivation key ──▶ output content hash

A derivation key contains every input the output varies on, and only those. Because it is built from content hashes, the same inputs produce the same key on every machine, which is what lets a build cache travel.

Filesystem paths, checkout roots, mtimes, and machine-local shop paths are not semantic inputs. Freshness is therefore a question over the same graph: walk the actor’s closure, read each source, derive the exact per-file key for the requested platform, and ask the catalog whether its output exists. The checks fan out in parallel.

closure(locator) ──▶ parallel(unit derivation checks) ──▶ current true/false

pit shop status <locator>, pit forge seed --check, and pit cement --check ask this question. They do not replay file lists or compare sidecar stamps. A changed source or producer identity simply makes its next per-file lookup miss; nothing is marked stale and unrelated artifacts remain usable.

Profile-guided optimization

Instrumented Mach pools associate counters and events with sites in the final pool, and the pool’s origin map normalizes those observations back to stable mcode function and site IDs. Observations are actor-local while they are being collected.

Observations are recorded in mcode identity, not in the pool’s. A persisted profile object records the mcode hash and function/site ID for each observation, with the originating pool and realization hashes, target and workload provenance, the counters or distributions themselves, and the schema and collector stamps kept alongside as provenance. Keying on mcode is what lets a profile outlive the pool it was gathered from: recompile, re-link, re-place, and the observations still find their sites.

A later finalization can consume selected profile objects and use them for inlining, fusion, specialization, constant layout, hot and cold layout, identical-code folding, and even link-plan selection. The canonical mcode object stays the stable source-derived IR; consuming a profile changes the pool derivation without re-identifying its input mcode.

Profile data is compiler input, and authority stays elsewhere: a profile obtained from a player or another party is validated against its pool, mcode, and profile provenance before use. Value observations can contain workload or user data, so release workflows aggregate or scrub them before they reach a shipped finalization record. Player artifacts carry collection tables and profile objects only when a diagnostic build asks for them.