Compilation and Shipping

Getting from source to a running program has two stages, and the second one is where your decisions live. This page is about those decisions: how much code shares a pool, what that buys you, and what you ship at the end.

The two stages

source ──▶ mcode              per file, portable

mcode units ──▶ pool          linked, optimized, target-final

Compiling to mcode happens per source file. The result is portable: no target, no memory layout, no decisions about what sits next to what. A file compiles to the same mcode whether it is headed for a desktop or a handheld.

Linking mcode into a pool takes one or more units and finalizes them together — optimizing across them, resolving their internal references, and laying the result out for one exact target. This is where the choices are. A pool is mapped and read where it sits; nothing is unpacked into a per-load constant or object tree on the way in.

Everything is content-addressed along the way, so a file that hasn’t changed isn’t recompiled, and two packages using the same source share one stored result.

Measuring one source unit

pit compile profile runs a source locator through the same mcode_unit_result entry point used by a shop builder and reports the honest source-to-portable-mcode cost:

pit compile profile runtime/tests/vm_suite.ce
pit compile profile runtime/tests/vm_suite.ce --repeat 3

The authoritative wall sample calls mcode_unit_result directly and collects only clocks. The report also includes source bytes and lines, total and significant tokens, requirements-analysis counts, raw and optimized IR size, and portable-unit size. A separate diagnostic compile clocks tokenize, parse, requirements, fold, mcode generation, and streamline; the displayed residual is the exact atom mean minus those separately sampled phases, so it includes portable-unit construction, source hashing, and ordinary sample variance. Token/IR census and named streamline-pass timings are likewise kept out of the authoritative wall sample.

Every registered streamline pass is enabled by default. Use the same per-compile controls as the shop’s diagnostic build lane to make A/B measurements:

pit compile profile runtime/tests/vm_suite.ce --disable-passes inline,sroa
pit compile profile runtime/tests/vm_suite.ce --all-passes-off
pit compile profile runtime/tests/vm_suite.ce --no-pass-timings

--stage also executes the content-addressed tail — canonical encode, content hash, and concurrent-safe write-once object staging — and reports it separately. It does not publish the locator’s catalog edge:

pit compile profile runtime/tests/vm_suite.ce --stage

How wide the fleet runs

A build compiles each unit of its closure in its own one-shot actor — one derivation, one actor — so a wide closure means many actors alive at once, each holding a compiler heap. That fleet is bounded by memory and by nothing else. There is no job-count limit: more builders than cores is the intended shape, because the scheduler already runs exactly one OS worker thread per core, and an extra alive actor costs a heap rather than a core.

The bound is a byte budget, a quarter of total system memory by default. A builder starts when its projected heap fits what the budget has left, and waits otherwise; the projection comes from the unit’s source size and is corrected upward whenever a builder is observed to peak above it. One job always runs even if it alone exceeds the budget — a fleet that admits nothing would stall forever — so a very large unit compiles on its own rather than not at all.

Override the budget with PIT_BUILD_MEMORY_BUDGET: a byte count, or none for an unbounded fleet.

PIT_BUILD_MEMORY_BUDGET=2147483648 pit <program>   # 2 GB of builders
PIT_BUILD_MEMORY_BUDGET=none       pit <program>   # no bound at all

A build request may also name an explicit width, which is a count ceiling on top of the byte budget; a fleet obeys both, and whichever binds first wins. Every plan’s telemetry reports what happened: the budget it ran under, the most it ever had admitted at once, how many builders waited for room, and the count ceiling if one was named.

This budget is not the per-actor heap limit. The heap limit is a ceiling on one actor, enforced inside the collector, which turns a single runaway into a named refusal. The budget is a fleet-wide admission gate in the shop, which decides how many actors may be alive at once. The limit bounds the tallest builder; the budget bounds their sum.

How much code per pool

The decision is how many mcode units go into each pool — the link plan — and it trades compile speed and reload granularity against optimization.

Those are the two shop forms, and there is no third: a cart is not a choice at all. A cart is inseparable from the binary it is embedded in, so it is pressed as one program — actors cross-linked, whole-program linked, fully optimized. It splits into several pools only for targets with ROM windows and paging.

The reason this matters beyond build time: a pool is the unit of replacement. Hot reload swaps a pool, so what you can reload without restarting is exactly what you put in its own pool. The link plan is therefore the hot-reload granularity lever as much as it is the optimization lever — and the two are one lever rather than two, because whole-program optimization folds distinct functions into one shared sequence, which is what stops {unit hash, function id} naming exactly one thing. A development build puts each module in its own pool for that reason; a shipped build merges, because nobody is reloading it and the optimizer does better with more to look at.

Pools are also the unit of memory layout, which is separate from the unit of optimization. How many pools you make is one decision; which ones are resident together and for how long is another. On a desktop that second choice rarely matters. On a console with a fixed budget it is the whole design — you decide which pools are always resident and which are acquired for an actor’s lifetime.

Products

A product is a cart with an entrypoint, and nothing more. One command makes one:

pit product <locator> [--include <locator>,...] [--target <system>] [--out <path>]

The entrypoint is exactly one locator. Its closure is found, that closure is pressed as one cart, and a binary is forged around the cart — so what comes out is a single executable that starts that actor on launch, on a machine that has never seen pit.

What defines a product is this invocation — a line in a Makefile. There is no product manifest and no [product] section in package.json: that file identifies behaviour for a collection of modules and programs, and which one of them you are shipping today is a build-time choice, not a property of the collection.

pit product is a formalization of two steps that already exist rather than a new mechanism: pit cement presses a cart and pit forge embeds it in a binary. ./pit itself is built exactly that way — it is the pit dev environment program, and its cart declares resident/shop/daemon where mcode’s declares one-shot/no-shop/no-daemon.

The closure includes the actors it can create

A program is not only its imports. If it calls $start, whatever it starts has to be in the cart too, or the shipped binary will ask for a program that is not there. So closure discovery walks $start sites: $start(callback, program) takes a locator as its second argument, every literal one is resolved and added to the closure, and the walk recurses into what it found — that actor’s imports and that actor’s own $start sites.

A second argument that is not a literal cannot be resolved, and is reported rather than dropped, with its file and source position:

  3 $start site(s) with a NON-LITERAL program argument — static analysis
  cannot name what they start. If any of them starts an actor that is
  not already in the list above, pass it with --include or the binary
  will refuse that start at run time:
    shop_tools/build.ce:127:9   $start(…, <not a literal>)

--include is how you answer that. --dry-run prints the whole report and presses nothing.

What a product declares about itself

A product declares itself one-shot, with no shop and no daemon. It boots in its own process, runs its root to completion, exits with that actor’s exit code, and the launcher creates nothing on disk — run it in an empty directory and the directory is still empty afterwards. --resident, --with-shop and --daemon say otherwise, one at a time, and a contradiction between them is refused by name rather than resolved by precedence.

Two shapes, chosen by what discovery found, not by a flag. A program that starts nobody is pressed with itself as the cart’s root: the smallest correct cart, with no shop actor in the image at all, so nothing in the process can start anything. A program that starts other actors is pressed with the root shop beside it, because something in the image has to serve a start from it.

Maximum optimization is the default

A product is pressed as one program, streamlined across its whole unit set, and stripped: producer names go, and so do the diagnostic log channels. What survives is console, error, panic and disrupt — the program’s own output and how it says it failed. A binary that could not name its own failure would be a worse program, not a smaller one. --no-strip keeps everything, which is what you want when you are debugging the product rather than shipping it.

None of this is needed in the dev loop, where it is understood that starting an actor may cause compilation.

Features are per product, not per target

Optionality belongs to the product. A feature is a name for a piece of the runtime that a program may not need — the daemon, $hook’s call and return hook — and turning one off does both halves of the job at once:

• The C goes. The feature’s package is swapped for its stub, so its translation units are never compiled, and a fold macro (PIT_HAVE_DAEMON, PIT_HAVE_HOOK) folds the caller-side sites away with it. $hook is the model: without the endowment, PIT_HOOK_ARMED is the constant 0 at every call and every return, so there is no ring, no field on the context, and no check in the dispatch loop.
• The shim goes. The provider module that made the feature’s endowment reachable from pit is dropped from the build, so nothing in the binary answers endowment/<name>.

The second half is what makes the first half safe. An actor that claims a name this product does not carry is refused by name at realization — told which endowment it asked for, which feature provides it, and that this program was built without that feature — rather than binding null and failing several frames later.

A product’s feature set is derived, never hand-listed. Two inputs:

1. The declaration. What the invocation says the program is already subtracts features. A product is one-shot with no shop and no daemon by default, and the cart records that in its start declaration; pit forge reads the bit back off the cart it is fusing in. So the daemon’s C leaves a one-shot product without anyone naming a feature. Because the build derives the fact from the artifact, the binary and the cart cannot disagree — and a --without daemon aimed at a cart that declares a daemon is refused rather than obeyed.
2. --without <feature>, for features no header bit names.

pit product pit-compiler/mcode.ce --without hook

An explicit opt-out that contradicts the closure is a build-time refusal, not a precedence question:

pit product: --without hook contradicts this product's own closure:
  shop_tools/debug.ce claims $hook, which the 'hook' feature provides
A product cannot both strip a feature and contain an actor that claims it.
Drop the --without, or leave that actor out of the product.

Letting the opt-out win would ship a binary that dies the first time that actor asks for the name; letting the closure win would ship a binary quietly containing what the invocation said to strip. Neither is what was asked for, so neither is chosen.

The features are listed in one table, shoplib/features.cm, and adding one is adding a row there plus a #ifndef PIT_HAVE_<X> default of 1 beside the C it guards. A feature is deliberately not a profile axis: stamping one would fork the realization cache and stop a dev-built cart mounting on the runtime that built it.

A recipe’s capabilities.daemon still says what a target is capable of. What it no longer decides is what one product carries.

Hot reload

In a development build, replacing code means building a new pool and installing it. The module body does not run again: a module’s export record is stone, so there is no re-initialization and no second record. The reloader walks the record the module already returned and repoints each exported function’s bytecode pointer at the new code. The actor keeps its state and its identity across the swap because nothing it holds is rebuilt.

Three things bound what a swap can do. A module can export a text, and a text cannot be reloaded — only functions have a pointer to repoint. The export shape must match: adding or removing a member is a warning or an error from the reload machinery, not a partial swap. And a function that is currently executing keeps running its old code and old frame layout until it returns; the repoint takes effect at the next call, so no effect is performed twice. The old pool stays pinned until the last frame using it unwinds.

Granularity follows directly from your link plan. If a module has its own pool, you can reload that module. If it was merged into a whole-program pool, reloading it means reloading everything merged with it. Choosing fine granularity while developing and coarse granularity when shipping is the normal arrangement.

Two ways to ship

Before choosing a form, there is a larger choice: are you shipping code onto someone else’s shop, or shipping the shop too?

Onto their shop. You publish a bundle or a cart, and it runs on whatever shop the recipient already has. Their shop decides what your code may hold, whether it can be compiled, and how patient it is with a slow actor. This is how you share a library or a tool with another developer.

With your own shop. You ship the runtime and a shop configured by you, and your program runs inside it. Now those decisions are yours: the shop you ship might carry no compiler, grant a small fixed set of endowments, and be as strict or as lenient about slow actors as your program needs. A game ships this way — the player is not running your code on a development machine, they are running the environment you built.

The second is what makes a shipped artifact predictable. Your development shop grants freely, compiles anything, and tolerates long turns because you are debugging. The shop you ship does not have to resemble it in any of those respects.

What comes along

A shipped shop carries the services your program needs and nothing else. You choose which actors to include: your own, plus whichever services those depend on. That choice is made per build, as the composing invocation’s own arguments — pit bundle … --payload mcode --with <actors> — with the standard invocations committed in the makefiles and scripts that build each product. It is not a property of the target, and no recipe holds it: the recipe describes the runtime binary, and the build description — the composer’s arguments — describes what rides on it.

The usual reason to include a service is a capability you want:

A program that renders a game and talks to nothing needs none of them. A moddable game needs the compiler and the policy actor, because that combination is what lets it accept new code and still decide what that code may do.

This is the only mechanism. What a shop can do is what its boot cart carries — there is no mode, no flag, and no switch to set. A build that cannot compile is one with no compiler actor in it; a build that cannot build C has no cake; a build that cannot resolve a name it was not shipped has no realizer. The absence is structural, so it can be checked rather than trusted. The debugging endowments settle the same way: a build links their providers or builds without them, and a claim with no provider in the binary is ungrantable.

It is also finer-grained than a mode could be. A cart with a pit compiler but no cake compiles mods and cannot touch C. A cart with a realizer but no compiler runs anything prebuilt it can find and builds nothing. Those are ordinary points in the space, not special configurations someone had to name.

If none of the combinations fit, write your own boot actor and put that on the cart. It is the first thing that runs and the thing that hands out authority, so a bespoke one is how you say something the standard pieces do not.

What you ship

Three forms, trading openness against size and speed.

A sealed ROM. The cart, the runtime binary, and the native code linked into one object, with every C binding resolved before shipping. Smallest and fastest — the layout is fixed and C calls bind directly — and not modifiable. This is the form for a console, and for any shipped game that has no reason to be opened.

A boot cart and a bundle. The cart is small and boots the runtime; the bundle carries the game. A minimal boot cart runs the bundle’s finalized pools directly, needing no compiler. The same bundle also carries source and the lockfile in its development slice, so a player with a development runtime can open it, change it, and rebuild. One artifact serves both.

A development cart and a bundle. The same arrangement with a cart built for debugging, so a program running on real hardware can be inspected from a development machine.

A cart is a sealed read-only store with a placement plan: its pools sit at final offsets, so mounting it is the whole of loading it. You choose what goes in — the actors it carries, the payload form they ship in, and the assets those actors need, from any package you depend on. A cart that carries a small compiler and its program’s source, compiling on the device at first boot, is the same tools and the same file; what a cart holds is the build description’s answer and not a fact about carts. What the bundle adds is portability — the same store snapshot with no placement, so it moves between machines. See The Store for how both are served, and Cartridges for the container format itself.

What is decided where

• You choose the link grouping, the placement, and the shipping form.
• The shop performs the compilation, caches by content, and assembles the result.
• The runtime maps pools and resolves the references between them — see Build and Artifacts for the machinery.