M2compiler-V3/docs/plan-two-phase.md

Two-phase refactor — analysis and plan (2026-10-02)

Decision: no restart. The AST/two-phase change is a middle-of-the- compiler refactor, not a rewrite. We reuse everything and convert the frontend's pass structure on a dedicated branch.

This document is the plan of record; docs/session-handoff-2026-10-01.md has the short version.

1. Why not restart from scratch

Everything painful in the last two sessions traced to one cause: lowering happens during parsing, so nothing can be looked at twice. An AST-first compiler fixes that — but a from-scratch restart would throw away the parts that are expensive and already correct:

Component	Reuse on restart?	Reuse on refactor?
Coco/R frame, scanner/parser driver (M2S/M2P/compiler.frm)	mostly	yes
SymTab — descriptors, scoping, checks, classes, vtables	large, hard-won	yes
QbeGen — emission, layouts, calls, sets, records, classes	the whole backend	yes
runtime shim, stdlib/, FileIO, fixpoint + corpus harnesses, tests	yes	yes
docs/ (language report, gaps, grammar)	yes	yes
AST.def/.mod (Stage A, tag v3-ast-stageA)	n/a	already started
M2.atg production grammar	yes (recognition is fine)	yes
M2.atg production actions	rewritten	rewritten (same grammar)

The one thing a restart legitimately replaces is the dialect/grammar — and the grammar is not the bottleneck. The pass structure is.

So: change what the grammar actions do (build nodes instead of emit), not what the grammar recognizes.

2. Pass structure (target)

per unit:
  1. Parse            -> AST (declarations + bodies), SymTab entries for
                         names/kinds as declared (order-independent later)
  2. Resolve/Check    -> bind identifiers, complete types, 2xx diagnostics
  3. Lower            -> walk AST, call the existing QbeGen API
  4. Emit             -> unchanged (buffered image, EndModule)

Key property: step 3 runs after the whole unit's declarations are in SymTab, so forward references resolve naturally. The text-patching machinery and the Design placeholder path are deleted.

3. Flag strategy (keep the suite green while converting)

• Add M2.atg flag TWO_PHASE (a grammar-level BOOLEAN or a QbeGen.SetLower switch).
• Phase 1: parser both builds AST nodes and emits (today's behaviour). AST building is inert; nothing reads it. Suite/fixpoint stay green. (This is basically Stage A applied to the productions.)
• Phase 2: implement Lower(n); run it in a test-only mode (compile a unit, lower from the AST, compare to the old .ssa).
• Phase 3: flip a unit's lowering to the AST walk; keep the old path for units not yet converted. Convert production-by-production.
• Phase 4: when everything lowers from the AST, delete the old actions and the patching code.

4. Increment order (branch ast-stage-c)

Suggested slices, each ending in build.sh + run_tests.sh + fixpoint.sh (revert the slice if either fails):

1. Flag + scaffolding — DONE (branch ast-stage-c, docs/summary_two-phase-slice1.md): twoPhase flag; AST linked into the compiler build; the integer leaf also builds NkIntLit. Nothing consumes it. Green (178/178, fixpoint OK).
2. Expressions → AST — PARTIAL (branch ast-stage-c): docs/summary_two-phase-slice2.md (spine, literals, parens, unary minus via astCur + per-invocation astIsLit), docs/summary_two-phase-slice3.md (Design: identifiers, [i]/.f/^ selectors, qualified names), and docs/summary_two-phase-slice4.md (calls/NkCall with actuals, NOT, builtins as named NkCalls). Still NoNode: brace/set literals, ResultComp suffixes; Lower + .ssa compare not started. Green (178/178, fixpoint OK).
3. Statements → AST — PARTIAL (branch ast-stage-c, docs/summary_two-phase-slice5.md): StatSeq builds NkBlock; assignment/call, IF/ELSIF/ELSE, WHILE, REPEAT, LOOP/ EXIT, FOR, RETURN, HALT build nodes. Deferred: CASE, WITH, builtin statements (INCL/INC/NEW/…), brace/set literals, ResultComp, and Lower. Green (178/178, fixpoint OK). Finding: local arrays in a PROCEDURE are miscompiled (docs/wip/local-array-bug.mod) — pre-existing, worked around here.
4. Declarations → AST — PARTIAL (branch ast-stage-c, docs/summary_two-phase-slice6.md, docs/summary_two-phase-slice7.md): DeclSeq builds NkDeclSeq; CONST/TYPE/VAR/PROC build NkConstDecl/NkTypeDecl/NkVarDecl/NkProcDecl (types referenced by the node ty index); astUnit holds NkUnit(name, decls, body) for all three unit kinds; builtin statements build NkCalls. Done. docs/summary_two-phase-slice8.md (CASE/WITH), docs/summary_two-phase-slice9.md (brace/set literals), docs/summary_two-phase-slice10.md (ResultComp + unbounded NkBlock/NkDeclSeq), docs/summary_two-phase-slice11.md (imports), docs/summary_two-phase-slice12.md (nested modules) and docs/summary_two-phase-slice13.md (CLASS, + a TypeBlock collection bug fix). Remaining: Lower + byte-compare, and the MaxChild cap on CASE arms / WITH designators / call actuals.
5. Flip: bodies lower from the AST; delete FwdPatchAll / FixLoadClass / FixStoreClass, the Design forward-variable placeholder, and pendVar* if superseded.
6. Cleanup + docs: update docs/features.md, language-report.md, analysis_gaps.md; tag.

5. Verification gates

• Fixpoint is the hard gate: ./bootstrap/fixpoint.sh must print FIXPOINT OK (byte-identical stage2/stage3). The compiler compiles itself, so it exercises the new path on real, large input.
• Suite run_tests.sh (178/178 today) after every slice.
• Corpus (tools/v3-corpus/corpus.sh) — expect the undeclared identifier / forward-reference buckets to shrink once the AST lands; track the compile-OK counts.
• Byte-compare in Phase 2/3: lower-from-AST .ssa vs old .ssa for the test suite; they should match until a feature deliberately changes.

6. What this unlocks (delete list / new capability)

• Delete: QbeGen.FwdPatchAll, FwdDesignator, FwdMangle, FwdAddrOper, FwdPatch, FwdElemPatch, FixLoadClass, FixStoreClass, BufReplaceAll; the grammar's FwdVarNote/ FwdVarFlush/nFvarRefs; the Design forward branch and SymTab.FwdVar*; EXPR's IsFwdVar leniency.
• New capability, now natural: forward procedures without a DEFINITION; set/aggregate values in CONST (unblocks ChanConsts/StreamFile, parked in docs/wip/); statement-context result suffixes (F()[i] := x); GOTO/labels; typed CONST.

7. Risks and mitigations

• Self-hosting order constraints — the AST removes them, but during the transition M2.atg/QbeGen.mod must still compile under the current compiler; keep declaration-before-use until the flip.
• Two paths diverging — mitigate with byte-compare in Phases 2–3.
• Scope creep into Stage C in one go — do not. Slices above are deliberately one concern each.
• Fixpoint red for a long stretch — expected in Phase 4; keep the old path available until every unit lowers from the AST.

8. State / entry point

Branch off v3-session-2026-10-02 (03f250e). Baseline: suite 178/178, fixpoint OK (2,900,782 bytes). Start with §4 slice 1.