# Two-phase refactor — analysis and plan (2026-10-02)

Decision: **no restart.**  The AST/two-phase change is a **middle-of-the-
compiler** refactor, not a rewrite.  We reuse everything and convert the
frontend's *pass structure* on a dedicated branch.

This document is the plan of record; `docs/session-handoff-2026-10-01.md`
has the short version.

## 1. Why not restart from scratch

Everything painful in the last two sessions traced to one cause:
**lowering happens during parsing**, so nothing can be looked at twice.
An AST-first compiler fixes that — but a *from-scratch* restart would
throw away the parts that are expensive and already correct:

| Component | Reuse on restart? | Reuse on refactor? |
| --- | --- | --- |
| Coco/R frame, scanner/parser driver (`M2S`/`M2P`/`compiler.frm`) | mostly | yes |
| **`SymTab`** — descriptors, scoping, checks, classes, vtables | large, hard-won | yes |
| **`QbeGen`** — emission, layouts, calls, sets, records, classes | the whole backend | yes |
| runtime shim, `stdlib/`, `FileIO`, fixpoint + corpus harnesses, tests | yes | yes |
| `docs/` (language report, gaps, grammar) | yes | yes |
| **`AST.def/.mod`** (Stage A, tag `v3-ast-stageA`) | n/a | already started |
| `M2.atg` production **grammar** | yes (recognition is fine) | yes |
| `M2.atg` production **actions** | rewritten | rewritten (same grammar) |

The one thing a restart legitimately replaces is the **dialect/grammar**
— and the grammar is not the bottleneck.  The **pass structure** is.

So: change what the grammar *actions do* (build nodes instead of emit),
not what the grammar *recognizes*.

## 2. Pass structure (target)

```
per unit:
  1. Parse            -> AST (declarations + bodies), SymTab entries for
                         names/kinds as declared (order-independent later)
  2. Resolve/Check    -> bind identifiers, complete types, 2xx diagnostics
  3. Lower            -> walk AST, call the existing QbeGen API
  4. Emit             -> unchanged (buffered image, EndModule)
```

Key property: **step 3 runs after the whole unit's declarations are in
SymTab**, so forward references resolve naturally.  The text-patching
machinery and the `Design` placeholder path are deleted.

## 3. Flag strategy (keep the suite green while converting)

- Add `M2.atg` flag `TWO_PHASE` (a grammar-level BOOLEAN or a
  `QbeGen.SetLower` switch).
- Phase 1: parser **both** builds AST nodes **and** emits (today's
  behaviour).  AST building is inert; nothing reads it.  Suite/fixpoint
  stay green.  (This is basically Stage A applied to the productions.)
- Phase 2: implement `Lower(n)`; run it in a **test-only** mode
  (compile a unit, lower from the AST, compare to the old `.ssa`).
- Phase 3: flip a unit's lowering to the AST walk; keep the old path for
  units not yet converted.  Convert production-by-production.
- Phase 4: when everything lowers from the AST, delete the old actions
  and the patching code.

## 4. Increment order (branch `ast-stage-c`)

Suggested slices, each ending in `build.sh` + `run_tests.sh` +
`fixpoint.sh` (revert the slice if either fails):

1. **Flag + scaffolding**: `TWO_PHASE` flag; parser builds AST nodes for
   one trivial production (e.g. `Ident`/`NkIdent`) alongside emission.
   Nothing consumes them.  Green.
2. **Expressions → AST**: `Expr`/`SimExpr`/`Term`/`Fact` build
   `NkBinExpr`/`NkUnary`/`NkIdent`/literals, *in addition* to emitting.
   Add `Lower` for expressions, exercised in a test-only compare.
3. **Statements → AST**: `Block`/`Stat` build `NkAssign`/`NkIf`/`NkWhile`/
   …; `Lower` them.
4. **Declarations → AST**: `ConstBlock`/`TypeBlock`/`VarBlock`/
   `ProcDecl`; the unit's decls complete SymTab before any body is
   lowered.
5. **Flip**: bodies lower from the AST; delete `FwdPatchAll` /
   `FixLoadClass` / `FixStoreClass`, the `Design` forward-variable
   placeholder, and `pendVar*` if superseded.
6. **Cleanup + docs**: update `docs/features.md`, `language-report.md`,
   `analysis_gaps.md`; tag.

## 5. Verification gates

- **Fixpoint is the hard gate**: `./bootstrap/fixpoint.sh` must print
  `FIXPOINT OK` (byte-identical stage2/stage3).  The compiler compiles
  itself, so it exercises the new path on real, large input.
- **Suite** `run_tests.sh` (178/178 today) after every slice.
- **Corpus** (`tools/v3-corpus/corpus.sh`) — expect the `undeclared
  identifier` / forward-reference buckets to shrink once the AST lands;
  track the compile-OK counts.
- **Byte-compare** in Phase 2/3: lower-from-AST `.ssa` vs old `.ssa` for
  the test suite; they should match until a feature deliberately
  changes.

## 6. What this unlocks (delete list / new capability)

- Delete: `QbeGen.FwdPatchAll`, `FwdDesignator`, `FwdMangle`,
  `FwdAddrOper`, `FwdPatch`, `FwdElemPatch`, `FixLoadClass`,
  `FixStoreClass`, `BufReplaceAll`; the grammar's `FwdVarNote`/
  `FwdVarFlush`/`nFvarRefs`; the `Design` forward branch and
  `SymTab.FwdVar*`; `EXPR`'s `IsFwdVar` leniency.
- New capability, now natural: forward **procedures** without a
  DEFINITION; **set/aggregate values in `CONST`** (unblocks
  `ChanConsts`/`StreamFile`, parked in `docs/wip/`); statement-context
  result suffixes (`F()[i] := x`); `GOTO`/labels; typed `CONST`.

## 7. Risks and mitigations

- **Self-hosting order constraints** — the AST removes them, but during
  the transition `M2.atg`/`QbeGen.mod` must still compile under the
  *current* compiler; keep declaration-before-use until the flip.
- **Two paths diverging** — mitigate with byte-compare in Phases 2–3.
- **Scope creep into Stage C in one go** — do not.  Slices above are
  deliberately one concern each.
- **Fixpoint red for a long stretch** — expected in Phase 4; keep the
  old path available until every unit lowers from the AST.

## 8. State / entry point

Branch off `v3-session-2026-10-02` (`03f250e`).  Baseline: suite 178/178,
fixpoint OK (2,900,782 bytes).  Start with §4 slice 1.