M2compiler-V3/docs/plan-separate-compilation.md

Plan — separate compilation for V3 (future work)

Status: deferred. Finish the compiler as it is first (corpus, language gaps, robustness). This document records the design so the work can be picked up later without re-deriving it.

Baseline at the time of writing: v3-smallfix; suite 259/259, fixpoint OK 4,028,447 bytes, corpus 511/612, 0 crashes.

---

1. What separate compilation means

        compile time                         link time
  .def (interface)  ──►  type checking
  .mod (body)       ──►  codegen  ──►  .o  ──┐
                                            ├──► cc/ld  ──► executable
  runtime shim + libc ──────────────────────┘

• Every compilation reads the .def of every imported module (transitively, because a def can re-export types from another def). It does not read the imported module's .mod, .o, or any previous compilation state.
• A .def change requires recompiling its importers (like a C header change). No link-time type checking.
• The linker receives only .o files, the runtime shim and system libraries. .def files are never passed to the linker.

2. Why V3 cannot do it today

V3 is a single-session, whole-program compiler:

• M2.mod parses all units on the command line into one SymTab, then Lower.EndSession calls QbeGen.EndModule, which writes one gen_ssa/<program>.ssa.
• Procedure symbols are <base>_<uid>, where uid comes from SymTab.nextUid in session parse order (QbeGen.Mangled). Separate compilations assign different uids to the same procedure → undefined symbols at link time.
• Vtable symbols are $vt_<typeIndex> (QbeGen.VtRef); type indices are session-local too.
• Literal/constructor pools are session-global: $strN, $setN, $ctoN.
• QbeGen.EndModule flushes all pools and emits main; there is no per-module object boundary.
• Module init bodies already emit <module>_init (QbeGen.BeginInit) and main calls the init functions encountered in the same session. Cross-object init ordering is the one new runtime concern.
• Generics re-parse the generic body in the client session (M2.mod DoInst).

3. Required design

3.1 Stable symbol names (critical path)

Entity	Current	Needed
Module-level procedure	<base>_<uid>	<module>__<name>
Module-level variable	<module>_<name> (already stable)	keep
Nested procedure / method	<base>_<uid>	<module>__<path> or deterministic local id
Class vtable	$vt_<typeIndex>	<module>__<class>__vt
String literal	$strN	<module>__strN
Set constant	$setN / named <unit>_<name>	module-qualified; exported consts keep global names
Constructor	$ctoN	<module>__ctorN
External C procedure	C link name	keep

Module-level names are unique in Modula-2; module qualification is enough. Nested procedures need a deterministic path (not the session counter). Generics use the declared instance name, so <instance>__<proc> is stable.

3.2 Per-unit codegen

• M2 -c Foo.mod emits one QBE module for Foo with export function for public procedures and data for globals.
• Library modules emit no main; only the program module emits main, the init calls and the ExitCode return.
• Split QbeGen.EndModule into “end object” and “end program”.
• Pools are per-object; exported set/string CONSTs are materialised by the client from the .def value or referenced by stable symbols.

3.3 Module initialization

• Each module with a BEGIN body/class init exports <module>_init (already the case).
• The program object's main calls the init functions of all linked modules once, in dependency order, before the program body.
• Simplest implementation: the driver/M2make computes the import DAG and emits an array of init function pointers in the program object.
• Nested modules are initialised by their enclosing module's init.
• Alternative: emit constructors into .init_array (needs C/asm glue; less portable with QBE). Start with the generated list.

3.4 Interface resolution / driver

• Add -c (compile one unit) and -I <dir> search paths.
• Read the target .mod plus every direct/transitive .def; never the imported .mod.
• Keep the current one-shot multi-file mode for tests/corpus.
• Optional .sym cache: serialise parsed def symbol tables, keyed by def hash + compiler version + flags. Not required initially; stale caches are a real risk.

3.5 Generics

• ISO generic modules are compile-time templates. A client instantiation still needs the generic body, not just the .def.
• Choose one:
  • instantiate into the client object (generic .mod must be on the search path), or
  • emit a separate instance module/object (Inst.o) with stable instance names and link it.
• The .def alone is enough for type checking, not for emitting code.

3.6 Classes / sets / strings

• Class layouts come from the .def; vtables/methods live in the defining module's object under stable names.
• Exported set/string CONSTs can be re-materialised from the .def value or referenced by a stable exported symbol.
• Avoid cross-object dependencies on private literal pools.

4. The linker

Normal cc/ld; no compiler-specific linker needed:

M2 -c -I runtime/syslib:lib/iso Lib.mod  -o Lib.o
M2 -c -I runtime/syslib:lib/iso Prog.mod -o Prog.o
cc Prog.o Lib.o runtime/syslib/shim.o -lm -o Prog

• .o files + shim + libc; no .def at link time.
• main only in the program object.
• Lib__P, Lib__x, Lib__Class__vt are ordinary symbols.
• Lib__init is exported and called from Prog's main via the generated init list.
• Undefined Lib__P at link time = stale object or .def/object mismatch → rebuild the importer.
• Runtime (shim.c, -lm) is linked once, as today.
• No linker script needed initially.

5. Phases (do not start until the current compiler work stops)

1. Stable names + per-unit .ssa in one session. Keep the whole-session driver, emit one .ssa per unit, link them. Validates symbols, pools, vtables and the init list.
2. -c / -I driver mode. Compile one .mod against transitive .defs; emit .o; link with cc.
3. Init order + program/main split. Generated init table, module bodies, nested modules.
4. Generics and classes. Instance emission strategy; stable vtable/method symbols.
5. M2make incremental builds. Per-module .o targets, .def dependencies, relink.
6. Self-host with separate compilation. The compiler as N objects; adjust the fixpoint gate (compare per-object .ssa or the linked binary).

6. Tests / gates for each phase

• A two-module program (Lib.def/.mod + Prog.mod) compiled as separate objects, linked, run.
• Globals, exported procedures, set/string CONSTs, class vtable dispatch, generic instances.
• Module init order (an init that depends on an imported module's init).
• Recompiling only the program after an unrelated module change.
• Recompiling an importer after a .def change.
• cc link failure on a stale object is expected and documented.
• The existing suite/fixpoint/corpus gates must stay green; the fixpoint may need a new comparison for multi-object builds.

7. Risks

• Stable naming is the highest-risk item: it touches imports, generics, classes, vtables, bootstrap and self-hosting.
• Generics need the generic implementation at client compile time; there is no “precompiled generic interface” shortcut for the body.
• Init order must be deterministic and dependency-aware.
• ABI/versioning: objects from different compiler versions/flags must not be mixed silently. Record a compiler/ABI version in the object or build system.
• Fixpoint: the “byte-identical single .ssa” gate must be redefined for multi-object builds.
• Debugging: separate objects make it easier to inspect a module, but harder to reproduce whole-program diagnostics until the driver is solid.

8. Non-goals / keep it simple

• No dynamic linking, no incremental type checking beyond .def mtimes, no linker plugin, no binary .def format initially.
• Keep .def as the source of truth; a .sym cache is optional.
• Keep the one-shot mode for the test suite/corpus.

Appendix — related future work: dialect library folders

The question “can PIM, M2R10 and ISO libraries live in separate folders?” is independent of separate compilation, but it compounds the same interface-resolution work. A possible layout:

runtime/syslib/     SysShim, FileIO, Utf8 (shared runtime)
lib/pim/            InOut, RealInOut, NumberIO, FpuIO, StrIO, ...
lib/iso/            STextIO, SWholeIO, SRealIO, TextIO, IOChan,
                    StreamFile, ChanConsts, IOConsts, SIOResult,
                    ConvTypes, Conversions, RealStr, WholeStr,
                    LongStr, LongMath, Strings, Storage,
                    ProgramArgs, SysClock, TERMINATION, ...
lib/m2r10/          future R10 redesigned library (not ISO/PIM)
lib/gm2/ (optional) gm2-only modules (DynamicStrings, Files, ...)
lib/common/         modules genuinely identical across dialects

Rules:

• One dialect per compilation session. PIM and ISO can both define InOut/RealInOut/Storage with different signatures; never put both folders on the same -I path for one session.
• The build system (M2make) selects lib/pim or lib/iso (or lib/m2r10), plus runtime/syslib.
• M2R10 is a different language revision, not a dialect of ISO/PIM; its library must not be mixed with either.
• The corpus harness currently uses one V3_LIBS; it would need to compile PIM and ISO corpora with different library paths (or audit that no colliding module is used across both).
• Keep stdlib/ as a compatibility aggregate only during the transition; remove it once the test/corpus scripts use the dialect folders.
• Classification of SysIO, Files, DynamicStrings, CharClass, WrapC/WrapClock/WrapTime, TimeString needs an audit: ISO, gm2-only, or V3-specific.

This is a low-risk directory/build change once the compiler is stable; it should be scheduled after the current corpus/language work, not during it.