Lower phase — L7 (the flip): status and plan (2026-10-05)

Branch ast-stage-c. L7 = "flip: lower from the AST instead of inline; delete the inline emit + Fwd* machinery".

Why L7 is not incremental

The flip makes Lower the sole emitter. The obstacle is that QbeGen holds one session state, used by the inline emit during parse; a per-unit Lower emit would be clobbered by the next unit's inline emit. So the flip is a big-bang: either remove/suppress the grammar's 533 QbeGen.* call sites, or give QbeGen two states — neither is a byte-verifiable step.

The pragmatic path is therefore a session re-emit: collect every unit's AST and re-emit the whole session with Lower after parsing. That is byte-comparable against the legacy session image and becomes the flip once the inline emit is removed.

Started (`d30abfd`, tag `v3-lower7-scaffold`)

Lower.LowerSession/LowerOneUnit/MaterializeGlobals.
Driver (compiler.frm) collects GetUnit() per file and calls Lower.LowerSession under -lower.
SymTab.Materialize makes a unit's own VAR/CONST visible in globScope, so GlobalRef resolves "<unit>_<name>" at session end (the unit scopes are closed by then).
Removed Lower.Reset from the M2 root so the owned tables persist across a session.

Still scaffolding (inline emit untouched); suite 200/200, FIXPOINT OK (3,401,129 bytes).

Progress (`ce2fc67`, tag `v3-lower7-session`)

DefUnit now builds an AST (proc headings recorded as NkProcDecl); LowerDecls replays a bodyless proc as a heading.
Session re-emit works for small sessions: lower_ok_files added, and DBasicProg, DQualProg, TQual (def-CONST), TQualP (def-PROC) byte-match. Fixes: image named after the program module; MaterializeGlobals also materializes TYPE/CLASS names; SymTab.ReenterModGlobal keeps nested-module globals' "<outer>_<name>" symbols; AST.Init once per session under -lower.

Suite 204/204; FIXPOINT OK (3,405,813 bytes).

Remaining gaps

AST-arena size (fundamental). LowerSession needs every unit's AST alive at once, but AST.MaxNodes = 65536 is sized for one unit; the compiler's session overflows it (MakeNode → NoNode, later units dump (nil)). Growing the arena past the 4 MiB sessBuf is not viable. So the session re-emit cannot handle the compiler — the flip must instead emit per unit at each unit's END (AST valid, scope open), which is the two-state / inline-emit-removal problem.
External procs. storage_prog differs: Lower emits $malloc_6(l link, ...) where the legacy emits $malloc(...) — the ext flag in Lower.NoteProc is hardcoded FALSE and Mangled's ProcLink fails after scopes close.
Definition-module proc replay (done), nested globals (done).

The "hang" was a Lower bug, not a miscompile

LowerDecls emitted a procedure's parameter loop even when ProcInfo failed, using an uninitialised np — so the loop ran ~forever inside FindProc. (MaxProc = 64 is far too small for a whole session, so ProcInfo failed for most of the compiler's procs.) Fixed (a9a8ecd): the proc is skipped unless ProcInfo succeeds, and each ParamInfo is guarded. Also grew the AST arena/pool (caeeeb5) — the arena/pool data lines are compact z runs, so this is nearly free.

Hash index attempted — also trips V3 miscompiles

Implemented an open-addressing name index (HashName/HashGetSym/ HashGetProc) for FindSym/FindProc in Lower. Results:

With the index, the session re-emit on the compiler's sources became fast (MaxProc=512 completed in <300 s; before it hung).
But the tables must hold the whole session (the compiler has 713 functions, so MaxProc ≥ 713), and at MaxProc ∈ {768,1024} the same run is slow/segfaults (CopySpan via ParamInfo) — the size/layout-sensitive bug again.
Worse, the hash itself makes the self-host fail: with the index in Lower, m2s2 (built by m1s1) produces no output on the compiler input (stage 3). So the index triggers another latent V3 miscompile.

Reverted to the green state (a9a8ecd). Suite 204/204.

Bisecting the hash miscompile

Reproduced with a proc-only hash (HashName/ProcEq/HashPutProc/ HashGetProc; FindSym left linear): the fixpoint still fails at stage 3.

m2s2 (V3-built) works on small programs (t_exit, t_arith, t_lower5set, t_classmethod) but aborts (SIGABRT) on Lower.def + Lower.mod.
Backtrace: HashPutProc ← NoteProc ← ProcHeading ← DefUnit — i.e. m2s2 crashes while parsing the definition module's proc headings.
The obvious cause — (hashProc[s2] = 0) OR ProcEq(VAL(CARDINAL, hashProc[s2]-1), name) evaluating the RHS (so 0-1 → huge index) — was restructured away, and a ProcEq bounds guard added; neither fixed it. So it is a deeper V3 miscompile of HashPutProc.

Narrowing further needs many ~10-minute fixpoint cycles; reverted to green (a9a8ecd). Suite 204/204.

Progress: per-unit flip unblocked (2026-10-05, after `a3ef897`)

The bad=T that blocked per-unit emission was a phantom: it was the pre-existing invalid call errors in Lower.mod (forward references), now fixed by declaring the walkers in Lower.def. With that gone, the per-unit flip works end to end.

Mechanism

Driver: Lower.BeginSession (open + SetNoEmit), BeginUnit (SaveSession) before each Parse, EndSession (write image).
Lower.LowerUnitNow(unit, isProg) after each Parse: RestoreSession → SetNoEmit(FALSE) → LowerOneUnit → SetNoEmit(TRUE).
QbeGen.EndModule is a no-op under NoEmit, so the grammar's unit-end emit neither flushes nor writes.
The grammar's suppressed inline emit must be a faithful shadow of the legacy emit: LocAdd/FuncParam no longer skip NewTemp under NoEmit, so the temp/NoteAddr numbering aligns with Lower's emit.

Name resolution

Whole-session tables: MaxSym=4096, MaxProc=2048, MaxPar=32.
Open-addressing hash index (HashStr/SpanEq/HashPut/HashGet, HashSize=8192) for FindSym/FindProc. The compiler session now lowers in ~1.5 s. (The earlier "hash miscompile" was a symptom of the miscompiled Lower; it is gone now that Lower is V3-legal.)
CopySpan's source is a VAR open array: gm2 copies a value open-array formal to a stack VLA of its full length, which overflowed the stack for the multi-megabyte tables.
External procs: Lower.MarkProcExternal (called from the grammar's EXTERNAL branch) records ext; LowerCall binds external callees to their own name (no uid, no static link).
Indirect calls through procedure-typed variables (CurrentCh): LowerCall detects a proc-typed VAR/PARAM callee and uses CallBeginInd.

Grammar AST fixes (also affect the two-phase AST generally)

ActParam saves/restores the enclosing call's astArgs/astNArgs: Factor reset astNArgs for designator arguments, so any call with two or more designator actuals kept only the last one.
ImplUnit/ProgModule reset astStmt := AST.NoNode after DeclSeq: the module-body child was the stale last procedure body, so modules without a module BEGIN (SysShim) grew a spurious _init.
Lower.NoteConstVal: const values are per-unit (every module has EOL), so they live in Lower's tables, not SymTab (which collided across modules after EndUnit).
LEN/HIGH added to LowerBuiltinExpr.

Remaining gap

The compiler session now lowers, but is not byte-identical to the legacy image yet: the first divergence is in M2S.Get ($Get_61), where Lower emits 148 labels vs. the legacy 430 — a large part of the Get body (a big CASE ch OF …) is missing. lower_ok single-file tests and the suite are green; the fixpoint is green (-lower is not used there).

Suite 204/204; FIXPOINT OK (3,695,508 bytes), session 0 errors.

Progress: more Lower coverage (`132a981`, tag `v3-lower7-gaps`)

First divergence moved to AST.AddTxt (line ~6961). Fixes:

CASE chunking. The arm list was capped at AST.MaxChild (8): CaseStat/CaseAlt appended arms with SetChild(cnode, NChild, arm), which silently drops beyond 8. They now build a chunked arm sequence (AstAppend, container kind NkBlock), and LowerCase follows the continuation chunks. This was the big M2S.Get gap.
VAL(T, e) in LowerBuiltinExpr, same widen/narrow/convert chain as the grammar.
Qualified calls (FileIO.ReadBytes) resolve through Lower's proc tables (SymTab's scopes are closed); CalleeAlias returns the base name so the mangled symbol stays <base>_<uid>.
VAR actuals. LowerExprFull records value→address with NoteAddr, mirroring the grammar's Design; AddrOfVal could not rely on the grammar's notes because its 256-entry ring wraps during a whole-session emit.
Bare proc values (CurrentCh := CharAt): Resolve recognises the proc table and LowerProcAddr emits the code address; sibling method names fall through to the THIS-binding path.

Next blocker: symbol scoping

Lower's symbol table is a single flat, latest-wins table, so common names (s, i, k) collide across procedures. E.g. HIGH(s) in AST.AddTxt (an open-array parameter) resolved to a fixed array of 256 from another procedure. lower_ok tests are too small to collide.

Two ways forward:

record a scope id per symbol and resolve within the active procedure chain, or
run Lower with SymTab's scope open (the grammar-based call) and resolve locals via SymTab first — the earlier attempt only broke EndModule's ExitCode lookup, fixable by materialising the program globals in EndSession.

Suite 204/204; FIXPOINT OK (3,711,028 bytes), session 0 errors.

summary_lower_l7.md 9.8 KB Link permanente Histórico Em bruto