# Lower phase — L7 (the flip): status and plan (2026-10-05)

Branch **`ast-stage-c`**.  L7 = "flip: lower from the AST instead of
inline; delete the inline emit + `Fwd*` machinery".

## Why L7 is not incremental

The flip makes `Lower` the sole emitter.  The obstacle is that `QbeGen`
holds **one** session state, used by the inline emit *during* parse; a
per-unit `Lower` emit would be clobbered by the next unit's inline emit.
So the flip is a **big-bang**: either remove/suppress the grammar's 533
`QbeGen.*` call sites, or give `QbeGen` two states — neither is a
byte-verifiable step.

The pragmatic path is therefore a **session re-emit**: collect every
unit's AST and re-emit the whole session with `Lower` after parsing.
That is byte-comparable against the legacy session image and becomes the
flip once the inline emit is removed.

## Started (`d30abfd`, tag `v3-lower7-scaffold`)

- `Lower.LowerSession/LowerOneUnit/MaterializeGlobals`.
- Driver (`compiler.frm`) collects `GetUnit()` per file and calls
  `Lower.LowerSession` under `-lower`.
- `SymTab.Materialize` makes a unit's own VAR/CONST visible in
  `globScope`, so `GlobalRef` resolves `"<unit>_<name>"` at session end
  (the unit scopes are closed by then).
- Removed `Lower.Reset` from the `M2` root so the owned tables persist
  across a session.

Still scaffolding (inline emit untouched); suite **200/200**, FIXPOINT OK
(3,401,129 bytes).

## Progress (`ce2fc67`, tag `v3-lower7-session`)

- **`DefUnit` now builds an AST** (proc headings recorded as
  `NkProcDecl`); `LowerDecls` replays a bodyless proc as a heading.
- **Session re-emit works** for small sessions: `lower_ok_files` added,
  and `DBasicProg`, `DQualProg`, `TQual` (def-CONST), `TQualP`
  (def-PROC) byte-match.  Fixes: image named after the program module;
  `MaterializeGlobals` also materializes TYPE/CLASS names;
  `SymTab.ReenterModGlobal` keeps nested-module globals'
  `"<outer>_<name>"` symbols; `AST.Init` once per session under `-lower`.

Suite **204/204**; FIXPOINT OK (3,405,813 bytes).

## Remaining gaps

1. **AST-arena size (fundamental).**  `LowerSession` needs every unit's
   AST alive at once, but `AST.MaxNodes = 65536` is sized for one unit;
   the compiler's session overflows it (`MakeNode` → `NoNode`, later
   units dump `(nil)`).  Growing the arena past the 4 MiB `sessBuf` is
   not viable.  So the session re-emit **cannot** handle the compiler —
   the flip must instead emit **per unit** at each unit's END (AST valid,
   scope open), which is the two-state / inline-emit-removal problem.
2. **External procs.**  `storage_prog` differs: Lower emits
   `$malloc_6(l link, ...)` where the legacy emits `$malloc(...)` — the
   `ext` flag in `Lower.NoteProc` is hardcoded `FALSE` and `Mangled`'s
   `ProcLink` fails after scopes close.
3. Definition-module proc replay (done), nested globals (done).

## The "hang" was a Lower bug, not a miscompile
`LowerDecls` emitted a procedure's parameter loop even when `ProcInfo`
failed, using an uninitialised `np` — so the loop ran ~forever inside
`FindProc`.  (`MaxProc = 64` is far too small for a whole session, so
`ProcInfo` failed for most of the compiler's procs.)  **Fixed**
(`a9a8ecd`): the proc is skipped unless `ProcInfo` succeeds, and each
`ParamInfo` is guarded.  Also grew the AST arena/pool (`caeeeb5`) — the
arena/pool data lines are compact `z` runs, so this is nearly free.

## Hash index attempted — also trips V3 miscompiles
Implemented an open-addressing name index (`HashName`/`HashGetSym`/
`HashGetProc`) for `FindSym`/`FindProc` in `Lower`.  Results:

- With the index, the session re-emit on the compiler's sources became
  **fast** (`MaxProc=512` completed in <300 s; before it hung).
- But the tables must hold the whole session (the compiler has **713**
  functions, so `MaxProc ≥ 713`), and at `MaxProc ∈ {768,1024}` the same
  run is **slow/segfaults** (`CopySpan` via `ParamInfo`) — the
  size/layout-sensitive bug again.
- Worse, the hash itself makes the **self-host fail**: with the index in
  `Lower`, `m2s2` (built by `m1s1`) produces no output on the compiler
  input (stage 3).  So the index triggers *another* latent V3 miscompile.

Reverted to the green state (`a9a8ecd`).  Suite 204/204.

## Bisecting the hash miscompile
Reproduced with a **proc-only** hash (`HashName`/`ProcEq`/`HashPutProc`/
`HashGetProc`; `FindSym` left linear): the fixpoint still fails at stage 3.

- `m2s2` (V3-built) **works on small programs** (`t_exit`, `t_arith`,
  `t_lower5set`, `t_classmethod`) but **aborts (SIGABRT)** on
  `Lower.def + Lower.mod`.
- Backtrace: `HashPutProc` ← `NoteProc` ← `ProcHeading` ← `DefUnit` —
  i.e. m2s2 crashes while parsing the definition module's proc headings.
- The obvious cause — `(hashProc[s2] = 0) OR ProcEq(VAL(CARDINAL,
  hashProc[s2]-1), name)` evaluating the RHS (so `0-1` → huge index) —
  was restructured away, and a `ProcEq` bounds guard added; **neither
  fixed it**.  So it is a deeper V3 miscompile of `HashPutProc`.

Narrowing further needs many ~10-minute fixpoint cycles; reverted to
green (`a9a8ecd`).  Suite 204/204.

---

## Progress: per-unit flip unblocked (2026-10-05, after `a3ef897`)

The `bad=T` that blocked per-unit emission was a **phantom**: it was the
pre-existing `invalid call` errors in `Lower.mod` (forward references),
now fixed by declaring the walkers in `Lower.def`.  With that gone, the
per-unit flip works end to end.

### Mechanism
- Driver: `Lower.BeginSession` (open + `SetNoEmit`), `BeginUnit`
  (`SaveSession`) before each `Parse`, `EndSession` (write image).
- `Lower.LowerUnitNow(unit, isProg)` after each `Parse`:
  `RestoreSession` → `SetNoEmit(FALSE)` → `LowerOneUnit` → `SetNoEmit(TRUE)`.
- `QbeGen.EndModule` is a no-op under `NoEmit`, so the grammar's unit-end
  emit neither flushes nor writes.
- The grammar's suppressed inline emit must be a **faithful shadow** of
  the legacy emit: `LocAdd`/`FuncParam` no longer skip `NewTemp` under
  `NoEmit`, so the temp/`NoteAddr` numbering aligns with Lower's emit.

### Name resolution
- Whole-session tables: `MaxSym=4096`, `MaxProc=2048`, `MaxPar=32`.
- Open-addressing hash index (`HashStr`/`SpanEq`/`HashPut`/`HashGet`,
  `HashSize=8192`) for `FindSym`/`FindProc`.  The compiler session now
  lowers in ~1.5 s.  (The earlier "hash miscompile" was a symptom of the
  miscompiled `Lower`; it is gone now that `Lower` is V3-legal.)
- `CopySpan`'s source is a **VAR** open array: gm2 copies a value
  open-array formal to a stack VLA of its full length, which overflowed
  the stack for the multi-megabyte tables.
- External procs: `Lower.MarkProcExternal` (called from the grammar's
  EXTERNAL branch) records `ext`; `LowerCall` binds external callees to
  their own name (no uid, no static link).
- Indirect calls through procedure-typed variables (`CurrentCh`):
  `LowerCall` detects a proc-typed VAR/PARAM callee and uses
  `CallBeginInd`.

### Grammar AST fixes (also affect the two-phase AST generally)
- `ActParam` saves/restores the enclosing call's `astArgs`/`astNArgs`:
  `Factor` reset `astNArgs` for designator arguments, so any call with
  two or more designator actuals kept only the last one.
- `ImplUnit`/`ProgModule` reset `astStmt := AST.NoNode` after `DeclSeq`:
  the module-body child was the stale last procedure body, so modules
  without a module `BEGIN` (SysShim) grew a spurious `_init`.
- `Lower.NoteConstVal`: const values are per-unit (every module has
  `EOL`), so they live in Lower's tables, not `SymTab` (which collided
  across modules after `EndUnit`).
- `LEN`/`HIGH` added to `LowerBuiltinExpr`.

### Remaining gap
The compiler session now lowers, but is **not byte-identical** to the
legacy image yet: the first divergence is in `M2S.Get` (`$Get_61`),
where Lower emits 148 labels vs. the legacy 430 — a large part of the
`Get` body (a big `CASE ch OF …`) is missing.  `lower_ok` single-file
tests and the suite are green; the fixpoint is green (`-lower` is not
used there).

Suite **204/204**; FIXPOINT OK (3,695,508 bytes), session 0 errors.

---

## Progress: more Lower coverage (`132a981`, tag `v3-lower7-gaps`)

First divergence moved to `AST.AddTxt` (line ~6961).  Fixes:

- **CASE chunking.**  The arm list was capped at `AST.MaxChild` (8):
  `CaseStat`/`CaseAlt` appended arms with `SetChild(cnode, NChild, arm)`,
  which silently drops beyond 8.  They now build a chunked arm sequence
  (`AstAppend`, container kind `NkBlock`), and `LowerCase` follows the
  continuation chunks.  This was the big `M2S.Get` gap.
- **`VAL(T, e)`** in `LowerBuiltinExpr`, same widen/narrow/convert chain
  as the grammar.
- **Qualified calls** (`FileIO.ReadBytes`) resolve through Lower's proc
  tables (SymTab's scopes are closed); `CalleeAlias` returns the base
  name so the mangled symbol stays `<base>_<uid>`.
- **VAR actuals.**  `LowerExprFull` records value→address with
  `NoteAddr`, mirroring the grammar's `Design`; `AddrOfVal` could not
  rely on the grammar's notes because its 256-entry ring wraps during a
  whole-session emit.
- **Bare proc values** (`CurrentCh := CharAt`): `Resolve` recognises the
  proc table and `LowerProcAddr` emits the code address; sibling method
  names fall through to the THIS-binding path.

## Next blocker: symbol scoping
`Lower`'s symbol table is a single **flat, latest-wins** table, so
common names (`s`, `i`, `k`) collide across procedures.  E.g. `HIGH(s)`
in `AST.AddTxt` (an open-array parameter) resolved to a fixed array of
256 from another procedure.  `lower_ok` tests are too small to collide.

Two ways forward:
1. record a scope id per symbol and resolve within the active procedure
   chain, or
2. run `Lower` with `SymTab`'s scope open (the grammar-based call) and
   resolve locals via `SymTab` first — the earlier attempt only broke
   `EndModule`'s `ExitCode` lookup, fixable by materialising the program
   globals in `EndSession`.

Suite **204/204**; FIXPOINT OK (3,711,028 bytes), session 0 errors.