# Two-phase refactor — analysis and plan (2026-10-02)

Decision: **no restart.**  The AST/two-phase change is a **middle-of-the-
compiler** refactor, not a rewrite.  We reuse everything and convert the
frontend's *pass structure* on a dedicated branch.

This document is the plan of record; `docs/session-handoff-2026-10-01.md`
has the short version.

## 1. Why not restart from scratch

Everything painful in the last two sessions traced to one cause:
**lowering happens during parsing**, so nothing can be looked at twice.
An AST-first compiler fixes that — but a *from-scratch* restart would
throw away the parts that are expensive and already correct:

| Component | Reuse on restart? | Reuse on refactor? |
| --- | --- | --- |
| Coco/R frame, scanner/parser driver (`M2S`/`M2P`/`compiler.frm`) | mostly | yes |
| **`SymTab`** — descriptors, scoping, checks, classes, vtables | large, hard-won | yes |
| **`QbeGen`** — emission, layouts, calls, sets, records, classes | the whole backend | yes |
| runtime shim, `stdlib/`, `FileIO`, fixpoint + corpus harnesses, tests | yes | yes |
| `docs/` (language report, gaps, grammar) | yes | yes |
| **`AST.def/.mod`** (Stage A, tag `v3-ast-stageA`) | n/a | already started |
| `M2.atg` production **grammar** | yes (recognition is fine) | yes |
| `M2.atg` production **actions** | rewritten | rewritten (same grammar) |

The one thing a restart legitimately replaces is the **dialect/grammar**
— and the grammar is not the bottleneck.  The **pass structure** is.

So: change what the grammar *actions do* (build nodes instead of emit),
not what the grammar *recognizes*.

## 2. Pass structure (target)

```
per unit:
  1. Parse            -> AST (declarations + bodies), SymTab entries for
                         names/kinds as declared (order-independent later)
  2. Resolve/Check    -> bind identifiers, complete types, 2xx diagnostics
  3. Lower            -> walk AST, call the existing QbeGen API
  4. Emit             -> unchanged (buffered image, EndModule)
```

Key property: **step 3 runs after the whole unit's declarations are in
SymTab**, so forward references resolve naturally.  The text-patching
machinery and the `Design` placeholder path are deleted.

## 3. Flag strategy (keep the suite green while converting)

- Add `M2.atg` flag `TWO_PHASE` (a grammar-level BOOLEAN or a
  `QbeGen.SetLower` switch).
- Phase 1: parser **both** builds AST nodes **and** emits (today's
  behaviour).  AST building is inert; nothing reads it.  Suite/fixpoint
  stay green.  (This is basically Stage A applied to the productions.)
- Phase 2: implement `Lower(n)`; run it in a **test-only** mode
  (compile a unit, lower from the AST, compare to the old `.ssa`).
- Phase 3: flip a unit's lowering to the AST walk; keep the old path for
  units not yet converted.  Convert production-by-production.
- Phase 4: when everything lowers from the AST, delete the old actions
  and the patching code.

## 4. Increment order (branch `ast-stage-c`)

Suggested slices, each ending in `build.sh` + `run_tests.sh` +
`fixpoint.sh` (revert the slice if either fails):

1. **Flag + scaffolding** — DONE (branch `ast-stage-c`,
   `docs/summary_two-phase-slice1.md`): `twoPhase` flag; `AST` linked
   into the compiler build; the `integer` leaf also builds `NkIntLit`.
   Nothing consumes it.  Green (178/178, fixpoint OK).
2. **Expressions → AST** — PARTIAL (branch `ast-stage-c`):
   `docs/summary_two-phase-slice2.md` (spine, literals, parens, unary
   minus via `astCur` + per-invocation `astIsLit`),
   `docs/summary_two-phase-slice3.md` (`Design`: identifiers,
   `[i]`/`.f`/`^` selectors, qualified names), and
   `docs/summary_two-phase-slice4.md` (calls/`NkCall` with actuals,
   `NOT`, builtins as named `NkCall`s).  Still `NoNode`: brace/set
   literals, `ResultComp` suffixes; `Lower` + `.ssa` compare not
   started.  Green (178/178, fixpoint OK).
3. **Statements → AST** — PARTIAL (branch `ast-stage-c`,
   `docs/summary_two-phase-slice5.md`): `StatSeq` builds `NkBlock`;
   assignment/call, `IF`/`ELSIF`/`ELSE`, `WHILE`, `REPEAT`, `LOOP`/
   `EXIT`, `FOR`, `RETURN`, `HALT` build nodes.  Deferred: `CASE`,
   `WITH`, builtin statements (`INCL`/`INC`/`NEW`/…), brace/set
   literals, `ResultComp`, and `Lower`.  Green (178/178, fixpoint OK).
   Finding: **local arrays in a PROCEDURE are miscompiled**
   (`docs/wip/local-array-bug.mod`) — pre-existing, worked around here.
4. **Declarations → AST** — PARTIAL (branch `ast-stage-c`,
   `docs/summary_two-phase-slice6.md`, `docs/summary_two-phase-slice7.md`):
   `DeclSeq` builds `NkDeclSeq`; `CONST`/`TYPE`/`VAR`/`PROC` build
   `NkConstDecl`/`NkTypeDecl`/`NkVarDecl`/`NkProcDecl` (types referenced
   by the node `ty` index); `astUnit` holds `NkUnit(name, decls, body)`
   for all three unit kinds; builtin statements build `NkCall`s.
   Done.  `docs/summary_two-phase-slice8.md` (`CASE`/`WITH`),
   `docs/summary_two-phase-slice9.md` (brace/set literals),
   `docs/summary_two-phase-slice10.md` (`ResultComp` + unbounded
   `NkBlock`/`NkDeclSeq`), `docs/summary_two-phase-slice11.md` (imports),
   `docs/summary_two-phase-slice12.md` (nested modules) and
   `docs/summary_two-phase-slice13.md` (`CLASS`, + a `TypeBlock`
   collection bug fix).  Remaining: `Lower` + byte-compare, and the
   `MaxChild` cap on `CASE` arms / `WITH` designators / call actuals.
5. **Flip**: bodies lower from the AST; delete `FwdPatchAll` /
   `FixLoadClass` / `FixStoreClass`, the `Design` forward-variable
   placeholder, and `pendVar*` if superseded.
6. **Cleanup + docs**: update `docs/features.md`, `language-report.md`,
   `analysis_gaps.md`; tag.

## 5. Verification gates

- **Fixpoint is the hard gate**: `./bootstrap/fixpoint.sh` must print
  `FIXPOINT OK` (byte-identical stage2/stage3).  The compiler compiles
  itself, so it exercises the new path on real, large input.
- **Suite** `run_tests.sh` (178/178 today) after every slice.
- **Corpus** (`tools/v3-corpus/corpus.sh`) — expect the `undeclared
  identifier` / forward-reference buckets to shrink once the AST lands;
  track the compile-OK counts.
- **Byte-compare** in Phase 2/3: lower-from-AST `.ssa` vs old `.ssa` for
  the test suite; they should match until a feature deliberately
  changes.

## 6. What this unlocks (delete list / new capability)

- Delete: `QbeGen.FwdPatchAll`, `FwdDesignator`, `FwdMangle`,
  `FwdAddrOper`, `FwdPatch`, `FwdElemPatch`, `FixLoadClass`,
  `FixStoreClass`, `BufReplaceAll`; the grammar's `FwdVarNote`/
  `FwdVarFlush`/`nFvarRefs`; the `Design` forward branch and
  `SymTab.FwdVar*`; `EXPR`'s `IsFwdVar` leniency.
- New capability, now natural: forward **procedures** without a
  DEFINITION; **set/aggregate values in `CONST`** (unblocks
  `ChanConsts`/`StreamFile`, parked in `docs/wip/`); statement-context
  result suffixes (`F()[i] := x`); `GOTO`/labels; typed `CONST`.

## 7. Risks and mitigations

- **Self-hosting order constraints** — the AST removes them, but during
  the transition `M2.atg`/`QbeGen.mod` must still compile under the
  *current* compiler; keep declaration-before-use until the flip.
- **Two paths diverging** — mitigate with byte-compare in Phases 2–3.
- **Scope creep into Stage C in one go** — do not.  Slices above are
  deliberately one concern each.
- **Fixpoint red for a long stretch** — expected in Phase 4; keep the
  old path available until every unit lowers from the AST.

## 8. State / entry point

Branch off `v3-session-2026-10-02` (`03f250e`).  Baseline: suite 178/178,
fixpoint OK (2,900,782 bytes).  Start with §4 slice 1.