M2compiler-V3/docs/plan-two-phase.md

Two-phase refactor — analysis and plan (2026-10-02)

Decision: no restart. The AST/two-phase change is a middle-of-the- compiler refactor, not a rewrite. We reuse everything and convert the frontend's pass structure on a dedicated branch.

This document is the plan of record; docs/session-handoff-2026-10-01.md has the short version.

1. Why not restart from scratch

Everything painful in the last two sessions traced to one cause: lowering happens during parsing, so nothing can be looked at twice. An AST-first compiler fixes that — but a from-scratch restart would throw away the parts that are expensive and already correct:

Component	Reuse on restart?	Reuse on refactor?
Coco/R frame, scanner/parser driver (M2S/M2P/compiler.frm)	mostly	yes
SymTab — descriptors, scoping, checks, classes, vtables	large, hard-won	yes
QbeGen — emission, layouts, calls, sets, records, classes	the whole backend	yes
runtime shim, stdlib/, FileIO, fixpoint + corpus harnesses, tests	yes	yes
docs/ (language report, gaps, grammar)	yes	yes
AST.def/.mod (Stage A, tag v3-ast-stageA)	n/a	already started
M2.atg production grammar	yes (recognition is fine)	yes
M2.atg production actions	rewritten	rewritten (same grammar)

The one thing a restart legitimately replaces is the dialect/grammar — and the grammar is not the bottleneck. The pass structure is.

So: change what the grammar actions do (build nodes instead of emit), not what the grammar recognizes.

2. Pass structure (target)

per unit:
  1. Parse            -> AST (declarations + bodies), SymTab entries for
                         names/kinds as declared (order-independent later)
  2. Resolve/Check    -> bind identifiers, complete types, 2xx diagnostics
  3. Lower            -> walk AST, call the existing QbeGen API
  4. Emit             -> unchanged (buffered image, EndModule)

Key property: step 3 runs after the whole unit's declarations are in SymTab, so forward references resolve naturally. The text-patching machinery and the Design placeholder path are deleted.

3. Flag strategy (keep the suite green while converting)

• Add M2.atg flag TWO_PHASE (a grammar-level BOOLEAN or a QbeGen.SetLower switch).
• Phase 1: parser both builds AST nodes and emits (today's behaviour). AST building is inert; nothing reads it. Suite/fixpoint stay green. (This is basically Stage A applied to the productions.)
• Phase 2: implement Lower(n); run it in a test-only mode (compile a unit, lower from the AST, compare to the old .ssa).
• Phase 3: flip a unit's lowering to the AST walk; keep the old path for units not yet converted. Convert production-by-production.
• Phase 4: when everything lowers from the AST, delete the old actions and the patching code.

4. Increment order (branch ast-stage-c)

Suggested slices, each ending in build.sh + run_tests.sh + fixpoint.sh (revert the slice if either fails):

1. Flag + scaffolding: TWO_PHASE flag; parser builds AST nodes for one trivial production (e.g. Ident/NkIdent) alongside emission. Nothing consumes them. Green.
2. Expressions → AST: Expr/SimExpr/Term/Fact build NkBinExpr/NkUnary/NkIdent/literals, in addition to emitting. Add Lower for expressions, exercised in a test-only compare.
3. Statements → AST: Block/Stat build NkAssign/NkIf/NkWhile/ …; Lower them.
4. Declarations → AST: ConstBlock/TypeBlock/VarBlock/ ProcDecl; the unit's decls complete SymTab before any body is lowered.
5. Flip: bodies lower from the AST; delete FwdPatchAll / FixLoadClass / FixStoreClass, the Design forward-variable placeholder, and pendVar* if superseded.
6. Cleanup + docs: update docs/features.md, language-report.md, analysis_gaps.md; tag.

5. Verification gates

• Fixpoint is the hard gate: ./bootstrap/fixpoint.sh must print FIXPOINT OK (byte-identical stage2/stage3). The compiler compiles itself, so it exercises the new path on real, large input.
• Suite run_tests.sh (178/178 today) after every slice.
• Corpus (tools/v3-corpus/corpus.sh) — expect the undeclared identifier / forward-reference buckets to shrink once the AST lands; track the compile-OK counts.
• Byte-compare in Phase 2/3: lower-from-AST .ssa vs old .ssa for the test suite; they should match until a feature deliberately changes.

6. What this unlocks (delete list / new capability)

• Delete: QbeGen.FwdPatchAll, FwdDesignator, FwdMangle, FwdAddrOper, FwdPatch, FwdElemPatch, FixLoadClass, FixStoreClass, BufReplaceAll; the grammar's FwdVarNote/ FwdVarFlush/nFvarRefs; the Design forward branch and SymTab.FwdVar*; EXPR's IsFwdVar leniency.
• New capability, now natural: forward procedures without a DEFINITION; set/aggregate values in CONST (unblocks ChanConsts/StreamFile, parked in docs/wip/); statement-context result suffixes (F()[i] := x); GOTO/labels; typed CONST.

7. Risks and mitigations

• Self-hosting order constraints — the AST removes them, but during the transition M2.atg/QbeGen.mod must still compile under the current compiler; keep declaration-before-use until the flip.
• Two paths diverging — mitigate with byte-compare in Phases 2–3.
• Scope creep into Stage C in one go — do not. Slices above are deliberately one concern each.
• Fixpoint red for a long stretch — expected in Phase 4; keep the old path available until every unit lowers from the AST.

8. State / entry point

Branch off v3-session-2026-10-02 (03f250e). Baseline: suite 178/178, fixpoint OK (2,900,782 bytes). Start with §4 slice 1.