M2compiler-V3/docs/STATUS.md

m2compiler-V3 — status (2026-10-06)

This answers "where are we after all the refactoring?".

TL;DR

• The compiler works and self-hosts today. run_tests.sh is 204/204 and bootstrap/fixpoint.sh is FIXPOINT OK (3,758,751 bytes, byte-identical stage2/stage3).
• The two-phase refactor (L0–L6) is done and committed. The last step, L7 "the flip", is in progress and is opt-in (-lower); it does not affect the normal build.
• The flip already produces the full compiler image (736/736 functions), but it is only ~74 % byte-identical to the legacy image: the first difference is at line 89,349 of 120,266.
• We are not near completion of the flip. The mechanism is done; what remains is closing the last code-generation gaps until the images are byte-identical. Each gap found so far has been a small, local fix, but the session is huge and byte-identity is exacting.

What the refactor actually is

There are two ways the compiler can turn Modula-2 into QBE IR:

1. Legacy "inline" emit — the grammar emits QBE IR while it parses. This is the default and is what the working compiler and the self-hosting fixpoint use.
2. Lower (two-phase) — the grammar builds an AST, and a separate module Lower walks the AST and emits QBE IR. This is selected with the -lower flag and is the goal of the refactor ("the flip" = make Lower the only emitter and delete the inline emit).

The refactor is being done in slices:

phase	what	state
L0–L2	scalars, control flow (IF/WHILE/FOR/…)	done
L3	procedures, calls, the self-host root-cause fix	done
L4	builtins, CASE, WITH	done
L5	records/arrays/pointers/sets/strings, literals	done
L6	nested modules, classes, qualified names	done
L7	the flip: Lower emits the whole session	in progress

Everything is committed in small steps; the many commits are incremental fixes to the new Lower path, not changes to the working legacy compiler.

Where exactly we are (L7)

The flip is driven per unit: the driver suppresses the grammar's inline emit (SetNoEmit), snapshots the QbeGen state before each unit's parse, restores it, and lets Lower re-emit the unit. On the compiler's own sources:

• -lower emits the full image: 736 functions (same as legacy), ~1.5 s.
• It is ~74 % byte-identical before the first divergence: first diff at byte 2,935,735, line 89,349 of 120,266.
• Progress of the first divergence across the session: line 9 → 327 → 1124 → 3437 → 6961 → 56294 → 70934 → 82194 → 86942 → 89349.

The first divergence is now the generic grammar ArgList call, which has 9 arguments but AST.MaxChild = 8 caps a call at 7, so Lower drops the tail.

What L7 fixed so far

• Lower is V3-legal (forward references declared in Lower.def; V3 has no whole-module pass) and the SymTab.FormalOk open-array def/impl bug.
• Whole-session tables + an open-addressing hash index for FindSym/FindProc.
• Symbol scoping: a procedure's locals no longer collide with another procedure's (HIGH(s) picked the wrong s).
• Chunked sequences where a list can exceed AST.MaxChild: CASE arms, multi-name VAR declarations, CASE bodies.
• External procedures, indirect calls (proc-typed variables), VAL, LEN/HIGH, qualified calls, qualified proc-typed variables, const values, 1-char strings passed as ARRAY OF CHAR.
• Per-unit global materialisation (MaterializeReplace).
• Duplicate nested procedure names resolve by scope and uid.

What remains

1. Calls with more than 7 arguments (the ArgList case): chunk the call actuals and make the argument loops follow the NkBlock chunks. (A prototype regressed AstCallNode's legacy emit and was reverted.)
2. Whatever the next first-divergence reveals. Historically each fix has exposed one more small gap; there may be a few more.
3. Only then: delete the grammar's inline emit (the literal flip) and remove the -lower flag.

Are we near completion?

• Of the working compiler: yes — it is done, green, and self-hosting. The refactor has not broken it.
• Of the L7 flip: the hard architectural part is finished (per-unit emit, scoping, tables, the AST chunking mechanism). The remainder is a tail of codegen gaps that only matter because the verification is exact byte-identity over a 120k-line session. Realistically a handful more small fixes; not a rewrite. It is not required for the compiler to function — it is a quality/architecture goal.

How to check for yourself

cd compiler
./build.sh          # rebuild
./run_tests.sh      # 204/204
cd ..
bootstrap/fixpoint.sh   # FIXPOINT OK