Working state — a note taken while the work happens, not a specification. The system as it is meant to be is in Architecture.

Can the compiler be broken apart? — seam investigation

Repo: /Users/johnalanbrook/Documents/work/cell/.claude/worktrees/agitated-dubinsky-a3501e Branch: claude/module-export-link-review-9f8813 (tip c13443301) Date: 2026-08-05. Code-reading only; no builds, no seeds, no gates run.

Serves plans/programs.md R3 (product = cart + entrypoint), R7 (compilation lanes = how many mcode units per mach pool), R8 (mcode is the only shared artifact; every platform lowers its own mach).

0. Headline

The three stages John names — source→mcode, mcode linking/LTO, and mcode→bytecode — are already three distinct bodies of code with mcode artifacts at every seam. Two of the three seams are clean today; one (compiler↔linker) is entangled by a single use() edge, and the entanglement is entirely about module closure, not about data or state.

Linking/LTO is not necessarily part of the mcode compiler. But the LTO passes are the compiler’s streamline, invoked by the linker as a library. So the honest factoring is not three products but three products over two code bodies: a front end, a shared optimizer, and a lowerer — where the optimizer ships with the linker and (today) also with the front end.

Two findings worth the headline:

• The mcode→bytecode product already exists and is already pit. mcode_lower.cm + mach_lower.cm + mach_pool_emit.cm read pit.mcode.unit@3 and write every byte of the pmp1 pool with zero compiler and zero linker code (§4). C only lowers mcode to an in-memory fragment record and reads pools; it has never written one.
• The one data gap is exports. The unit artifact has no export declaration, so the linker recovers export shape by escape analysis and degrades to opaque on six named conditions (§2.1, §6.1). That is the only place where “the linker needs the compiler’s knowledge” is true — and the fix is a field, not a merge.

1. The dependency graph

1.1 Package ownership

Note pit-linker/build.cm and toolchains.cm are the C linker (dylibs, QBE, target toolchains): build.cm:18-27 imports file, shoplib::path, process::run, cake::plan, shoplib::symbols. It shares the package name with mcode_link.cm and nothing else. They are already two products living in one directory.

1.2 use() edges inside pit-compiler

Front end (reachable only from compiler.cm’s first five imports):

compiler.cm  (pit-compiler/compiler.cm:1-12)
 ├→ tokenize.cm        (no use() at all)
 ├→ parse.cm           → numeric_literals, resolve          (parse.cm:1-2)
 │    └ resolve.cm     → numeric_literals                   (resolve.cm:39)
 ├→ fold.cm            → rep_fold, numeric_literals         (fold.cm:4-5)
 │    └ rep_fold.cm    → runtime::internal/sysinfo          (rep_fold.cm:39)
 ├→ mcode.cm           → resolve, analysis, numeric_literals(mcode.cm:1-3)
 ├→ analysis.cm        → diagnostics                        (analysis.cm:3)
 ├→ diagnostics.cm     (no use())
 ├→ json, blob         (std)
 ├→ shoplib::crypto                                          (compiler.cm:11)
 ├→ shoplib::mach       ← ONLY used by run_ast helpers        (compiler.cm:8)
 ├→ shoplib::mach_press ← ONLY used by run_ast helpers        (compiler.cm:9)
 └→ streamline.cm  ────────────────────────────────┐

Middle end (streamline.cm:8-26), a disjoint subtree:

streamline.cm
 ├→ ir_stats(→ json, stream_ir)   ir_stats.cm:10-11
 ├→ time (std)
 ├→ int_ranges (→ numeric_literals, stream_ir)   int_ranges.cm:6-7
 ├→ licm       (→ numeric_literals, stream_ir)   licm.cm:31-32
 ├→ escape     (→ stream_ir)                     escape.cm:29
 ├→ notstone   (→ stream_ir)                     notstone.cm:27
 ├→ sroa       (→ stream_ir)                     sroa.cm:43
 ├→ forward    (→ stream_ir)                     forward.cm:39
 ├→ guardcse   (→ stream_ir)                     guardcse.cm:41
 ├→ tco        (→ callresolve, stream_ir)        tco.cm:44-45
 ├→ callresolve(→ stream_ir)                     callresolve.cm:46
 ├→ inline     (→ stream_ir, callresolve)        inline.cm:4-5
 ├→ passflags  (→ runtime::internal/os)          passflags.cm:27
 ├→ switch     (→ numeric_literals, stream_ir)   switch.cm:80-81
 ├→ record_shapes (→ stream_ir)                  record_shapes.cm:8
 ├→ numeric_literals (no use())
 ├→ stream_ir  (no use())
 ├→ stone_records (→ stream_ir)                  stone_records.cm:21
 └→ panic_outline (→ stream_ir)                  panic_outline.cm:10

Verified fact: nothing in the streamline subtree imports tokenize, parse, fold, mcode, resolve, analysis, or diagnostics. The optimizer does not know the front end exists. (Full use() census: grep -rn "use(" pit-compiler pit-linker — every hit is listed above or is a diagnostic string.)

Not in compiler.cm’s closure at all: structured.cm (no imports; used by mcode_link + tests), pgo.cm (pgo.cm:8-11; used by pit-shop/shop_build.cm:10 and pit-shop/pgo_store.cm:8), lto.cm (tools/LSP only), compact_spine.cm, ast_find.cm, cross_unit_inline_experiment.cm. verify_ir.cm and analyze.cm have no importer anywhere in the tree (grep for use( naming them returns nothing).

1.3 pit-linker → pit-compiler

pit-linker/mcode_link.cm:14-16:

def compiler  = use('pit-compiler::compiler')
def structured = use('pit-compiler::structured')
def json = use('json')

static_link.cm has no use() at all. build.cm/toolchains.cm do not import the compiler.

Everything the linker takes from compiler (exhaustive — grep -n "compiler\." pit-linker/mcode_link.cm):

All five are exported at compiler.cm:1783-1810. None of them is a front-end function. The linker never calls tokenize, parse, fold, mcode, analyze, mcode_unit_result, or compile_result.

So the linker’s semantic dependency is on {codec, validator, streamline}. Its module-closure dependency is on all of pit-compiler, because use() is eager and compiler.cm is the module it imports.

1.4 The lowering path

pit-shop/mcode_lower.cm      → shoplib::mach_press               (mcode_lower.cm:25)
pit-shop/mach_lower.cm       → shoplib::mach, mcode_lower,
                               shoplib::mach_pool_emit,
                               shoplib::mach_press,
                               runtime::internal/sysinfo,
                               shoplib::canonical, build_telemetry (mach_lower.cm:148-154)
shoplib/mach_press.cm        → shoplib::mach, shoplib::mach_pool_emit,
                               runtime::internal/sysinfo, canonical   (mach_press.cm:31-34)

No file in the lowering path imports pit-compiler or pit-linker. mcode_lower.cm:14-18 states this as policy:

“This module is deliberately an INDEPENDENT reader of pit.mcode.unit@3: it does not import the compiler, because the shop actor’s start path must not pull the whole compiler closure in behind it. That is what a portable artifact is for — the producer and the reader agree on bytes, not on shared code.”

That is a working, shipped, independent implementation of the mcode→mach consumer. It duplicates compiler.cm:1093-1170 (unit_target_code / target_function_from_portable / inline_unit_literals) as mcode_lower.cm:263-286 / :197-209 / :170-193. The duplication is deliberate and is the proof that seam 2 is already cut.

1.5 The whole pipeline as it runs (press lane)

pit-shop/boot_cart.cm:118-194 pool_executable:

exe.modules[].mcode.unit           (pit.mcode.unit@3, N of them)
  → link_rows_for(exe)                                     boot_cart.cm:87-103
  → mcode_link.link_result(rows, PRESS_LINK_PLAN)          boot_cart.cm:128
        ├ export_summary / import-edge finalization        mcode_link.cm:2546-2602
        ├ import_cross_unit_bodies (CP2 body copy)         mcode_link.cm:2612
        ├ streamline_linked_units → compiler.streamline
        │    {profile:"ship", stage:"link", …}             mcode_link.cm:2424-2478
        └ pool_program_literals (program-wide literal pool) mcode_link.cm:2626
     ⇒ pit.mcode.program@1
  → mcode_link.materialize_unit(program, unit_id)          boot_cart.cm:157
     ⇒ pit.mcode.unit@3 again (per member, group literals reattached)
                                                            mcode_link.cm:1657-1679
  → mcode_lower.lowering_code(unit)                        boot_cart.cm:167
  → mach_lower.press_unit(unit, locator, stamp, code)      boot_cart.cm:176
        ├ mach.mach_compile_mcode_bin(...{fragment:true})  mach_lower.cm:343-347   [C]
        └ pool_emit.emit([{fragment, unit, …}], {target})  mach_lower.cm:376-378   [pit]
     ⇒ pool bytes

Dev lane (no linker at all — boot_cart.cm:34-36: “The dev lane never reaches this file”):

shop_build.finish_unit_mach     pit-shop/shop_build.cm:934
  → shop_build.lower_mach_unit  pit-shop/shop_build.cm:759-784
      → build_fleet.request({type:"mach", mcode_hash, label, numrep, endian})
          ⇒ crosses an ACTOR boundary carrying ONLY a content hash
      → pit-shop/builder_worker.ce  "mach" op
          → mcode_lower.lowering_code → mach_lower  (builder_worker.ce:36-37)

2. What crosses each boundary as data

2.1 source → mcode: pit.mcode.unit@3

Normative spec: docs/spec/artifact-formats.md:448-657 (“Portable mcode unit”). docs/spec/mcode.md:27-35 covers the instruction language and states mcode itself has no constant pool — @3 interns on the way out. docs/architecture/pipeline.md:10-25 is the chain listing (the closest thing to a schema registry; there is no registry file — searched).

Producer: compiler.mcode_unit_result (pit-compiler/compiler.cm:1641-1711). Second producer (link-time re-encode): portable_unit_from_code (compiler.cm:1448-1458, stamps schema at :1450) — this is the boundary the linked streamline returns through (mcode_link.cm:2468). Top-level fields, as literally constructed at :1666-1707:

Is it self-sufficient for a linker? YES. Evidence:

1. mcode_link.link_result takes rows = [{unit_id, locator, unit, imports}] and a plan (mcode_link.cm:2482). It reads nothing else. The one thing it asks the compiler for is the codec and the optimizer — code, not state.
2. pit-shop/mcode_lower.cm:221-258 lowering_valid_unit validates the unit with exact-keys discipline listing the same 13 fields and no others (:222-227), and lowering_code (:263-286) reconstructs the full generator view from the record alone. An independent reader already exists and works.
3. The shop’s covering/manifest reads unit.imports and unit.claims straight out of the artifact — pit-shop/shop_build.cm:1140-1142, pit-shop/executable_manifest.cm:353-359, pit-shop/shop_source.cm:1879,1894.

Side-channels that are NOT in the artifact (things a consumer must be told separately):

• The link plan (pit.mcode.link-plan@1) — not in any unit; a constant at boot_cart.cm:43-48, validated at mcode_link.cm:73. Correct: it is a policy input, not producer data.
• The import covering — rows[].imports. boot_cart.cm:79-86 documents that the press passes imports: [] because the press compiles on the static link lane, so the units contain no import op. The covering is decided by the shop, not carried in the unit. This is the one real “the artifact does not say it” input to linking.
• locator — passed alongside (boot_cart.cm:96), not in the unit. compiler.cm:1685-1688 says this is deliberate: “Canonical package/file provenance is deliberately absent … the executable manifest binds the unit to where it came from.”
• The compiler identity for cache keys — opts.compiler_identity is accepted by mcode_unit_result but explicitly kept OUT of the bytes (compiler.cm:1672-1679).
• numrep — a compile-time input (builder_worker.ce:60, compiler.cm:1592 → fold_mod(…, {numrep})). See §2.4 defect.
• The unit’s own content hash — not a field. The store address IS the name (pit-shop/shop_store.cm:776-779); cached_mcode_artifact re-attaches it out of band as {unit, hash} (:782-785). tests/executable_claims.ce:295 asserts the record must not carry it.
• The live compiler identity — compiler.identity is the STABLE tag hash("compiler-v3"); the invalidating identity lives only in the derivation key shop_store.mcode_cache_key (shop_store.cm:97-105).

The one field that is genuinely MISSING from the artifact: exports. There is no exports field anywhere in pit.mcode.unit@3. The linker recovers export shape by CFG + two-pass escape analysis over the main function’s instruction stream — mcode_link.export_summary (mcode_link.cm:696-736, build_blocks :715, analyze_export_body :722/:734) — and it degrades to {kind: "generic"} for empty_body, body_too_large, unresolved_jump, no_return, multiple_returns, return_unreachable. So the compiler knows the export shape and throws it away; the linker reconstructs it lossily. This is the single largest data argument for keeping linking inside the compiler — and the fix is to add an exports field rather than to merge the products (see §6.1).

2.2 mcode → linked: pit.mcode.program@1

Producer: mcode_link.link_result (mcode_link.cm:2482-2649); the record is built at :2626-2647:

Spec: docs/spec/artifact-formats.md:538-617. Note the member unit inside a program is re-stamped pit.mcode.linked-unit@1 with literals: [] and group-indexed literal operands (mcode_link.cm:1474-1476) — deliberately not standalone (artifact-formats.md:585-596).

Is it self-sufficient for lowering? YES, and the code proves it. materialize_unit(program, id) (:1657-1679) reconstitutes a member as a standalone pit.mcode.unit@3 purely from program.units + program.groups, re-stamps schema, reattaches group.literals, and re-validates. The lowerer downstream (mcode_lower.lowering_code) then sees an ordinary unit and does not know a linker ran. boot_cart.cm:153-157 states exactly this.

The program record’s only compiler touch is compiler.schemas.mcode_unit (a string) and compiler.valid_mcode_unit at :1676-1678.

But the program is NEVER SERIALIZED OR STORED. The only non-test occurrences of SCHEMA_PROGRAM are the five inside pit-linker/mcode_link.cm; no store, catalog, or nota path handles it. It exists only as an in-process value between link_result and materialize_unit (boot_cart.cm:128 → :157 in the same function). Consequence for a three-product split: see §5.2b.

Spec drift worth noting (documented, not a defect to fix here): stats, cross_unit_inline_origins, plan.join_types, plan.inline_budget, units[].imports and member_imports[].provider_unit are all produced by the code and absent from the spec example at artifact-formats.md:545-570. The @3 example at :455-505 also shows stages: ["parse","fold-exact",…] and flags: {compact_slots, infer_shapes} where the code emits ["parse","fold","mcode","streamline"] (compiler.cm:27) and {compact_slots: true, streamline: true} (compiler.cm:1683).

2.3 linked → mach: what the lowerer needs beyond the unit rows

mach_lower.press_unit(unit, label, stamp, code) (mach_lower.cm:336-385) and fragment_unit(unit, label, code, opts) (:184-241) need:

No literals table, claims table, or endowment table is needed beyond the unit. claims is consumed by the shop (covering/manifest, executable_manifest.cm:353-359), never by the lowerer — mcode_lower.lowering_valid_unit requires it present (:225-227) but lowering_code never reads it. imports IS read, by the pool emitter.

So the only semantically load-bearing side-channel at this seam is the target stamp — which is correct and unavoidable: the artifact is target-neutral by design, so the target has to come from somewhere else. Everything else the lowerer needs is in the unit or derived from it.

2.4 The one real hole in “mcode is portable” (already ruled OPEN, 7.3)

tokenize.cm:280-285 builds every number token with number: number(raw) — the host’s parse. fold.cm:117-119 (make_number) re-renders folded results as {value: text(val), number: val} — host arithmetic, host value. compiler.cm:1210-1217 (literal_decimal) then spells that host value with json.encode so it round-trips exactly on this host.

Result: an mcode unit produced on a nan64 host embeds nan64’s limits (e.g. 1e300 → null). rep_fold.cm (fold.cm:4, rep_fold.cm:39) handles the cross-rep folding case but only when opts.numrep names a different rep, and it PUNTS rather than fixing the tokenizer’s eager parse. This is plans/programs.md §7.3 and it is confirmed by the code. It does not block splitting the products — it blocks shipping mcode between reps.

3. Where LTO lives

CONFIRMED. mcode_link.link_result → streamline_linked_units (mcode_link.cm:2613) → compiler.streamline(code, null, {profile, stage: "link", inline, inline_budget, join_types}) at mcode_link.cm:2461-2464.

The LTO passes are the compiler’s own optimizer, invoked by the linker as a library, on IR the linker decoded with the compiler’s own codec (compiler.unit_target_code :2438) and re-encoded with the compiler’s own encoder (compiler.portable_unit_from_code :2468).

What is genuinely the linker’s code, not the compiler’s:

• import-edge finalization / export_summary / mutable_export_risk (mcode_link.cm:696, :2546-2568)
• cross-unit body import (CP2 phase 3) — import_cross_unit_bodies (:2612); the actual splice is then done by the compiler’s inline pass
• program-granularity literal pooling (pool_program_literals :1433-1493)
• stripping (strip_program :1600)
• materialize_unit (:1657)

3.1 Could a standalone linker ship streamline without the front end?

Structurally, yes. The evidence:

1. The streamline subtree is a closed set of 19 modules that import only each other, stream_ir, numeric_literals, json, time, and runtime::internal/os (§1.2). Zero front-end edges.
2. stage: "link" is a first-class, already-shipped concept: passflags.cm:113 PASSFLAGS.stages = {unit: true, link: true}; passflags.cm:197-209 selects profile[stage] and applies link_passes/unit_passes independently; DEV_LINK is a distinct profile (passflags.cm:92-94).
3. stream_ir.cm is a pure codec over flat parallel arrays with a static opcode table (stream_ir.cm:21-53) and no cross-call mutable state — its only interning is opcode text, “interned once per actor” (:19). There is no shared interned-IR pool between the front end and the optimizer. CP1’s interning is inside one streamline() call: the module “converts that public form once at its entry edge … and restores canonical rows once at its return edge” (stream_ir.cm:4-6).
4. Diagnostics: the linker builds its own diagnostic records (mcode_link.cm:44-55 diagnostic(code,message,unit,member)) and never imports pit-compiler::diagnostics. No entanglement.

The one blocker is a packaging fact, not a code fact: use() is eager and module-granular, so use('pit-compiler::compiler') pulls tokenize+parse+fold+mcode+analysis+resolve+diagnostics into the linker’s closure. compiler.cm:1-12 is the whole of the problem.

3.2 The measured cost of that edge

plans/archive/night-2026-08-04/press-switch.md:110-132 reports the press-switch cart delta: two NEW sections, pit-compiler/structured (147.4 KiB) and pit-linker/mcode_link (138.1 KiB), +292,352 bytes, versus −45,158 bytes of real savings from the linked press. The note says the compiler front end was already in the boot cart (87 references), so the front-end edge cost nothing there — but it would cost everything in a standalone mcode-link product.

plans/programs.md phase 2 already rules this: “This is also where the press moves out of the shop actor — pool_executable runs inside the shop actor whose closure is the cart, which is the entire +292 KB the press switch cost.”

3.3 The seam is already formalized elsewhere in the tree

pit-shop/builder_identity.cm:55-57:

def COMPILE_ENTRIES  = ["pit-compiler/compiler", "pit-shop/shop_store"]
def LOWERING_ENTRIES = ["pit-shop/mcode_lower", "pit-shop/mach_lower",
                        "pit-shop/shop_store"]

This computes two derivation identities by BFS over the executable manifest’s import edges (:114-151) so a compiler edit does not re-lower every pool and a press edit does not recompile every unit. The comment at :26-30 names the one remaining overlap: “shoplib::mach_press is currently shared only because compiler.cm still exports compile-and-run helpers”.

plans/archive/derivations.md:412-427 already proved that overlap is spurious:

“pit-compiler/compiler.cm imports shoplib::mach_press and shoplib::mach only for compile_to_blob, compile_pipeline, run_ast, and run_ast_noopt. The production builder’s compile/floor messages call mcode_unit_result and compile_result; neither reaches those helpers.”

Verified: grep -n "mach\.\|press\." pit-compiler/compiler.cm returns exactly three hits — :1736, :1750, :1751, all inside those four helpers (compiler.cm:1733-1765). Their callers outside the compiler are tests/lto.cm:53, tests/suite.cm:343, shop_tools/internal/diff_runner.cm:329,334,340, and shop_tools/internal/compiler_perf.cm — tests and tools only. No production path calls them.

4. Who emits mach bytes today

Verdict: pit writes every byte of the pool. C never writes one.

source/mach_pool.c:6-8 states the rule outright — “C READS A POOL AND NEVER WRITES ONE. The emitter is shoplib/mach_pool_emit.cm” (repeated at :42-48 and :3306). The only "pmp1" writes in C are test fixtures (source/mach_pool_test.c:306,619,704,805; source/resident_image_provider_test.c:66); every production C mention is a read-side memcmp (source/mach_pool.c:514, :3316, source/cart_boot.c:241, source/pit.c:118).

The pit writer is shoplib/mach_pool_emit.cm, write_pool (:659): file = blob.make() :738; the pmp1 magic :739-742; version/endian/flags/header_size/section_count/file_size :743-748; the 9-row section table :756-762; ENTRIES :766-774; FUNCTIONS :776-800; every 32-bit instruction word put32(file, e, instructions[i]) :813-818; VALUES :820-843; STONE :851-856; SHAPES/SHAPE_KEYS/IMPORTS/FUNCTION_REFS :861-911; payload checksum spliced over the zeroed field :923-928; returns {pool, pool_hash, content_hash, …} :930-940. The byte primitives are pit functions too (put8 :71, put16 :77, put32 :90, put64 :105, put_bytes :143). mach_pool_emit.cm:60-67: “BYTE ORDER IS THIS FILE’S BUSINESS AND NOWHERE ELSE’S.”

Two steps, and shoplib/mach_press.cm:7-19 is the seam that names them:

1. C lowers. source/mach_pool.c, reached through the shoplib::mach native module as mach.mach_compile_mcode_bin(name, code, {fragment: true, mach: false, target}) — mach_press.cm:120-121, mach_lower.cm:224-226, mach_lower.cm:343-347. C entry shoplib/mach.c:13 → pit_mach_compile_mcode_bin (source/runtime.c:8646) → Pit_MachPoolFragment (source/mach_pool.c:3087, returns :3297). Output is a pit record, schema: "pit.mach.fragment@1" (mach_pool.c:3131): “target-final constants (STONE object bytes, target immediate words) plus the only two things a linker cannot know — where the pool-global shape operands sit, and the mach-pc → mcode-instruction provenance column” (mach_lower.cm:10-14). runtime.c:8752-8758 explicitly refuses to hand back serialized bytes: “there is no serialized mach payload any more — ask for {fragment: true} and link it with shoplib::mach_pool_emit”. Superinstruction fusion happens here, over the in-memory tree (mach_pool.c:2148,2277,2291, called from runtime.c:8743) — not at file emission.
2. Pit lays out the pool. shoplib::mach_pool_emit.emit(rows, {target}) — mach_press.cm:132-134, mach_lower.cm:246, mach_lower.cm:376-378. mach_lower.cm:14-16: “shoplib::mach_pool_emit is then a pure linker, and what comes out is a pit.mach.pool@1 the start walk runs from WHERE IT LIES: mapped, not parsed”.

The rule is stated at mach_press.cm:17-19:

“Pit emits pools, C reads them, one writer per format — the C single-unit assembler died with phase 2a for exactly that reason.”

mach: false is passed everywhere in production (mach_press.cm:117-121, mach_lower.cm:221-226, :344-346) — the old pmac serializer is not even run.

4.1 What “a pit file that takes mcode and turns it into bytecode” still needs

It already exists, minus one C function. The chain mcode_lower.cm (325 lines of pit: reads pit.mcode.unit@3, no compiler dependency) → mach_lower.cm (391 lines of pit) → mach_pool_emit.cm (1,190 lines of pit: writes every pool byte) is the pit file John is describing. The only C left in it is mach_compile_mcode_bin — the mcode→fragment lowering: instruction selection, frame layout, target-final constant materialization, fusion, and the provenance column.

The one thing C contributes that ends up in the file is STONE text objects, built by Pit_MachStoneText (source/mach_pool.c:2690) and placed verbatim by pit at mach_pool_emit.cm:854 — pit asks for them at :233. Even the fragment’s u32 word blobs (mach_frag_words, mach_pool.c:2820) are little-endian by transport convention only; pit reads them back word by word (mach_pool_emit.cm:522) and re-writes them at the target’s byte order (:816).

plans/programs.md:256-260 rescopes old P5 (“the Pit lowering”) under R8: it is “now needed for exactly one thing — cross-target carts”, because every platform lowers its own mach with its own C. The foreign-target refusal at mach_lower.cm:205-219 is therefore “correct and permanent for every other path” — and the refusal text itself already names the two-call recipe for the cross-press that is supported: mach_compile_mcode_bin({fragment: true, target}) then mach_pool_emit.emit(rows, {target}) (mach_lower.cm:214-217).

So: you do not need a pit lowerer to get three products. The mcode→bytecode product exists today as mcode_lower + mach_lower + mach_pool_emit + the C floor. A pit rewrite of mach_compile_mcode_bin would make that product portable across targets, which is a different (and per R8, narrower) goal.

4.2 The two chains, verified end to end

Press/cart lane (boot_cart.cm:245,258,272,282 → pool_executable):

boot_cart.cm:118  pool_executable
 :157   mcode_link.materialize_unit(program, next_id)     ⇒ pit.mcode.unit@3
 :167   mcode_lower.lowering_code(unit)                   ⇒ generator view
 :176   mach_lower.press_unit(unit, locator, stamp, code)
          mach_lower.cm:340  target_of(stamp)                       [mach_lower.cm:304]
          mach_lower.cm:343  mach.mach_compile_mcode_bin(…fragment:true, mach:false, target)
            shoplib/mach.c:13 → source/runtime.c:8646 → :8766
              source/mach_pool.c:3087 Pit_MachPoolFragment  ⇒ RECORD, no bytes
          mach_lower.cm:376  pool_emit.emit([{fragment, unit, unit_id:0, imports:[], mcode_hash}], {target})
            mach_pool_emit.cm:1111 emit → :943 emit_inner → :1100 write_pool → :659
              :738 blob.make · :739 "pmp1" · :816 put32 per word · :930 return {pool,…}
 :188   copy.mach = {hash, blob: window, embedded: true}

Dev lane (shop_build.cm:1068 → finish_unit_mach):

shop_build.cm:934   finish_unit_mach
 :945   cache_mach_result → early return on derivation cache hit  (:946-951)
 :955   lower_mach_unit(cb, artifact.hash, locator, rep)          (:759-784)
 :763     build_fleet.request(cb, {type:"mach", mcode_hash, label, numrep, endian, cache_hit})
            build_fleet.cm:205 request → worker OR local
   worker:  builder_worker.ce:190 lowering_code · :192 fragment_unit · :197 link_fragments
   local:   build_fleet.cm:97      lowering_code · :100 fragment_unit · :106 link_fragments
 mach_lower.cm:184 fragment_unit → :224 mach_compile_mcode_bin (fragment only)
 mach_lower.cm:243 link_fragments → :246 pool_emit.emit → mach_pool_emit.cm write_pool
 mach_lower.cm:258 pool_emit.origin_columns (best-effort sidecar; producer :1135)
 build_fleet.cm:176 / builder_worker.ce:227  shop_store.stage_mach_result ⇒ mach_hash
 shop_build.cm:964  publish_mach_result

Note what crosses the actor boundary in the dev lane: a content hash and a rep, nothing else (shop_build.cm:778-783). The mcode→bytecode stage is already running as a separate process-like unit with an artifact-only interface.

5. The verdict, seam by seam

Seam A — source → mcode (front end alone)

Clean today, one extraction away from being a shipping product.

• Data out is self-contained (pit.mcode.unit@3, §2.1).
• The front-end subtree has no incoming edge from the optimizer or the linker.
• The only wrong edges are compiler.cm:8-9 (shoplib::mach, shoplib::mach_press), used solely by four test/tool helpers (compiler.cm:1733-1765).
• Decoupling needed: move compile_to_blob, compile_pipeline, run_ast, run_ast_with_options, run_ast_noopt into a new pit-compiler/runner.cm; retarget tests/lto.cm:53, tests/suite.cm:343, shop_tools/internal/diff_runner.cm:329,334,340, shop_tools/internal/compiler_perf.cm. ~35 lines moved, 5 call sites retargeted. Already proposed at plans/archive/derivations.md:423-427.
• Caveat, not a blocker: the front end still needs streamline (it runs stage:"unit" at compiler.cm:1616, :1745). So mcode the product is front end + optimizer, not front end alone. That is fine — it matches MCODE_STAGES = ["parse","fold","mcode","streamline"] (compiler.cm:27), which is baked into the artifact’s compiler.stages field.

Seam B — mcode → linked (the linker)

Needs one specific decoupling: split compiler.cm in two.

The linker needs {schemas.mcode_unit, valid_mcode_unit, unit_target_code, portable_unit_from_code, streamline}. All five live in compiler.cm alongside the front-end driver, so importing them drags tokenize/parse/fold/mcode/analysis/resolve/diagnostics in.

• Decoupling: extract the artifact codec + validators + the streamline re-export into pit-compiler/mcode_unit.cm (or a pit-mcode package): functions valid_mcode_unit (compiler.cm:991), unit_target_code (:1093), target_function_from_portable (:1153), inline_unit_literals (:1128), portable_facts (:1067), portable_fact_bucket (:1079), apply_facts (:943), select_facts (:914), portable_unit_from_code (:1448), the fact-key tables (:778-827), the literal pool (:1189-1315), portable_body (:1430), plus the schema constants (:16-18). compiler.cm then imports that module and re-exports for compatibility; mcode_link.cm:14 imports it directly.
• Size: ~450 lines moved out of compiler.cm’s 1,811, no logic change. There are no back-edges to move — everything in that set already depends only on json_mod (:1211) and pure helpers.
• After that, the standalone linker’s closure is: mcode_unit codec + streamline’s 19 modules + structured.cm + mcode_link.cm. Zero front-end code.
• No hidden state to untangle. stream_ir has no cross-call interning (§3.1.3); passflags is a stone table (passflags.cm:88-113); diagnostics are the linker’s own (mcode_link.cm:44-55).

5.2b — the second, smaller gap at this seam: the program has no file form

pit.mcode.program@1 is never serialized or stored (§2.2). For a three-binary CLI pipeline the linker’s output has to land somewhere. Two ways, both cheap:

• Encode it. shoplib::canonical (nota) encodes arbitrary records, so the format is free; what is missing is a store/catalog path and a reader. Cost is a stage_*/read_* pair plus a validator, ~100 lines.
• Or skip the program entirely at the product boundary. The linker product reads N pit.mcode.unit@3 files and writes N materialized pit.mcode.unit@3 files — materialize_unit (mcode_link.cm:1657-1679) already does exactly this conversion, and boot_cart.cm:157 already calls it immediately after linking. mcode in, mcode out, no new format at all. This is the option that matches R8 (“mcode is the only shared artifact”) literally, and it is close to zero work.

The second option loses cross-unit literal sharing across the boundary (each materialized unit gets its group’s whole literal table reattached at :1677), which matters for pool size but not for correctness. If that matters, the program record can stay in-process and the linker product can own the lowering call too — i.e. products 2 and 3 fuse, which is the shape boot_cart.pool_executable already has.

Seam C — mcode/linked → bytecode (the lowerer)

Already clean. Zero work.

• pit-shop/mcode_lower.cm is a complete, independent pit.mcode.unit@3 reader that explicitly refuses to import the compiler (:14-18).
• mach_lower.cm:148-154 and mach_press.cm:31-34 import no compiler code.
• The seam already crosses an actor boundary in the dev lane carrying only a content hash: shop_build.cm:759-784 sends {type:"mach", mcode_hash, label, numrep, endian} to builder_worker.ce.
• builder_identity.cm:55-57 already computes the lowering closure separately from the compile closure, and BOTH ops already advertise from one program purely as a packaging convenience (builder_identity.cm:15-17).
• Residual: builder_worker.ce:33 imports pit-compiler::compiler for its compile op (:72, :133) — a lowering-only product simply drops that import and the compile/floor message handlers.

6. Answering John directly

“Can these be cleanly separated, or is the linking and LTO necessarily part of the mcode compiler?”

Linking is not part of the mcode compiler. Import-edge finalization, export exactness, cross-unit body import, program literal pooling, stripping and materialize_unit are all mcode_link.cm’s own code with zero compiler involvement (§3).

LTO is not part of the mcode compiler either — but it is the same code as the per-unit optimizer. compiler.streamline runs at stage:"unit" for the front end and at stage:"link" for the linker; passflags has treated those as two peer stages since it was written (passflags.cm:113). That is a good design, not an entanglement: one optimizer, two scopes, one set of passes to maintain.

The practical consequence for R3 products:

Streamline is shared by the first two. If John wants three products with no shared pit code at all, the choice is: (a) accept that mcode and mcode-link both carry the optimizer (~19 modules), or (b) make the front-end product emit unoptimized mcode and let the linker own all optimization — which contradicts the artifact’s own compiler.stages declaration (compiler.cm:27, :1682) and would lose the per-unit facts the linked streamline re-seeds (compiler.cm:1063-1066: those facts “cannot be reconstructed by re-running streamline over already compacted code”).

Recommended factoring — three products over three code bodies:

1. pit-compiler = front end (tokenize/parse/fold/mcode/analysis/resolve/ diagnostics) — depends on 2.
2. pit-mcode (new, extracted from compiler.cm) = the pit.mcode.unit@3 codec + validators + the streamline subtree. Depends on nothing.
3. pit-linker = mcode_link + structured — depends on 2 only.
4. pit-lower = mcode_lower + mach_lower + mach_pool_emit — depends on neither 1 nor 2 nor 3. Already true today.

Total work to get there: the ~35-line runner extraction (seam A) plus the ~450-line codec extraction (seam B). No data format changes. No state to untangle. No new artifact — provided the linker product emits materialized pit.mcode.unit@3 rather than a serialized program (§5.2b).

6.1 The one change that would make the seam better, not just possible

Add an exports field to pit.mcode.unit@3.

Today the compiler computes export shape and discards it; the linker reconstructs it by escape analysis over the main function’s instruction stream (mcode_link.export_summary, mcode_link.cm:696-736) and gives up — {kind: "generic"} — on body_too_large, unresolved_jump, no_return, multiple_returns, return_unreachable (:722/:734). Every such giving-up is an LTO opportunity lost to a recovery limitation, not a semantic one.

This matters directly for R5 (module returns stoned at the module main’s return) and R4 (hot reload walks the export record and refuses on shape mismatch) — both want a declared export shape, and both are phases 3 and 4 of this same arc. One field serves the linker, the reloader, and the stone rule.

It also strengthens the product split: with declared exports, the linker needs strictly less compiler knowledge, not more.

7. Things worth flagging

• pit-compiler/verify_ir.cm (469 lines) and pit-compiler/analyze.cm (62 lines) have no importer anywhere in the tree. Dead weight in any product that ships the package wholesale.
• pit-linker mixes two unrelated products: mcode_link.cm (semantic linker) and build.cm+toolchains.cm (2,605 lines of C-toolchain driver). Splitting the package would make the mcode-link product’s closure obvious.
• The pit.mcode.unit@3 reader is implemented twice — compiler.cm:1093-1170 and mcode_lower.cm:170-286. Deliberate (documented at mcode_lower.cm:14-18), and it would become unnecessary if the codec were extracted to a dependency-free pit-mcode: both could import it and the independence property would be preserved by the module having no front-end edges rather than by duplication.
• §7.3’s rep taint (tokenize.cm:284, fold.cm:118) is the one thing that makes mcode not actually portable between numeric reps. It does not block the product split; it blocks R8’s “mcode is the one shared artifact” claim.

Source: plans/proposal-notes/compiler-seams.md