Jelajahi Sumber

docs: L3 option 2 reaches 183/183; nested-array-field bug found

Option 2 (zero SymTab surface) is functionally complete: t_lower3
byte-matches and the suite is 183/183.  Records the flat-table design,
the grammar hooks, ScopeLeave-as-no-op, and the two real bugs.

New finding (repro docs/wip/nested-array-field-bug.mod): a RECORD field
that is a nested ARRAY OF ARRAY OF CHAR, accessed arr[i].field[j][k],
miscompiles -> the generated program segfaults (139 vs 42); the flat
form works.  This crashed the V3-built compiler until the Lower tables
were flattened.

Self-host still fails: m2s2 crashes on the full compiler input (works on
small programs); shrinking the tables did not help, so it is a
size-triggered latent V3 codegen bug.  Next: fix the nested-array field
in QbeGen, then bisect the size-triggered crash.

Gates: suite 182/182; fixpoint OK (3147849 bytes).
Eric Streit 6 hari lalu
induk
melakukan
7c0d6c430e
2 mengubah file dengan 138 tambahan dan 65 penghapusan
  1. 77 65
      docs/handoff-l3-option2.md
  2. 61 0
      docs/wip/nested-array-field-bug.mod

+ 77 - 65
docs/handoff-l3-option2.md

@@ -1,84 +1,96 @@
-# Handoff — L3 via option 2: functionally proven, blocked by no-FORWARD
+# Handoff — L3 via option 2: 183/183 but self-host blocked by V3 codegen
 
-Branch **`ast-stage-c`**, HEAD **`90b1d76`**.  Suite **182/182**;
+Branch **`ast-stage-c`**, HEAD **`edf24d9`**.  Suite **182/182**;
 fixpoint **OK** (**3,147,849 bytes**).  Clean except `toto.mod`.
 
 ## Headline
 
-Option 2 (zero `SymTab` surface) **works functionally**: with it,
-`t_lower3.mod` re-emits **byte-identical** to the legacy image and the
-suite reaches **183/183**.  It does **not** self-host yet, and the cause
-is now precisely known and is **not** the table design.
+Option 2 (zero `SymTab` surface) is **functionally complete** and
+**byte-verified**: `t_lower3` matches the legacy image and the suite
+reaches **183/183** with `lower_ok t_lower3`.  It **still does not
+self-host**, and the cause is now a *different* V3 codegen bug, not the
+table design.
 
-## What was built and proven (this session, then reverted to keep green)
+## What was built and verified this session
 
 `compiler/src/Lower.def` / `Lower.mod`:
 
-- Owned tables: `syms[]` (`name/kind/ty`) and `procs[]`
-  (`uid/res/depth/ext/params`), with `Reset`, `NoteVar`, `NoteParam`,
-  `NoteProc`, `SetProcRes`, `ScopeEnter`/`ScopeLeave`, and resolvers
-  `FindSym`/`FindProc`/`Resolve`/`ProcInfo`/`ParamInfo`.  **No `SymTab`
-  edits at all.**
+- Owned, **flat** tables (no arrays of records — see the bug below):
+  `symName : ARRAY OF CHAR` (stride 64), `symKind`, `symTy`, and
+  `procs : ARRAY OF ProcEnt` with a flat `pname` (stride 64).  Helpers
+  `Reset`, `NoteVar`, `NoteParam`, `NoteProc`, `SetProcRes`,
+  no-op `ScopeEnter`/`ScopeLeave`, `CopyN`/`CopySpan`, `FindSym`/
+  `FindProc`/`Resolve`/`ProcInfo`/`ParamInfo`.  **No `SymTab` edits.**
 - All `SymTab.ProcUid/ProcNPar/ParamType/...` call-sites switched to
   `ProcInfo`/`ParamInfo`; `LowerDesign`/`IsSimpleScalar`/`IsCall` to
   `Resolve`/`FindProc`.
-- Grammar hooks (inert; fill tables only): `M2` root `Lower.Reset`;
-  `VarDecl`/`ConstDecl` `Lower.NoteVar`; `ProcHeading` `Lower.ScopeEnter`
-  + `Lower.NoteProc` + `Lower.SetProcRes`; `ParamSection`
-  `Lower.NoteParam`/`NoteVar`; `ProcDecl` `Lower.ScopeLeave` and
-  `AST.SetOp(astNode, 0)`.
-- **Key bug fixed:** `ScopeLeave` must be a **no-op during parse** — a
-  real pop removed locals/params *before* the end-of-unit walk (same
-  closed-scope trap as `SymTab`).  With it no-op'd, procedure bodies emit
-  and `t_lower3` matches.
-
-## The blocker (why it can't self-host yet)
-
-`Lower` now has **three mutually-recursive procedure pairs**:
-
-- `CanExpr` ⇄ `CanCallArgs`
-- `CanProc` ⇄ `CanDecls` (via `CanDecl`)
-- `LowerExpr` ⇄ `LowerCall`
-
-The V3 self-hosted compiler rejects mutual recursion **without `FORWARD`**
-(`Lower.mod:NNN invalid call`), and `FORWARD` is not usable either (it
-breaks gm2's whole-program pass — see `docs/summary_two-phase-lower2.md`).
-The committed `Lower.mod` avoids this by **merging** mutually recursive
-bodies into one dispatching procedure (`CanStmt`, `LowerStmt`).
-
-## The fix (next session)
-
-Break all three cycles the same way the committed code already does —
-**merge, don't split**:
-
-1. **Inline `CanCallArgs` into `CanExpr`**'s `NkCall` branch (it is a
-   few lines: arity + recurse `CanExpr` on actuals).
-2. **Inline `CanProc` into `CanDecl`** (call `CanStmt` on `NkProcDecl`
-   body/children directly).
-3. **Inline `LowerCall` into `LowerExpr`**'s `NkCall` branch.
-
-After inlining, no procedure calls a later or mutually-recursive one, so
-the V3 compiler accepts it and the fixpoint should return.  All three are
-mechanical (their bodies are small and already written).
+- Grammar hooks (inert): `M2` root `Lower.Reset`; `VarDecl`/`ConstDecl`
+  `Lower.NoteVar`; `ProcHeading` `Lower.ScopeEnter`+`NoteProc`+
+  `SetProcRes`; `ParamSection` `Lower.NoteParam`/`NoteVar`; `ProcDecl`
+  `Lower.ScopeLeave` + `AST.SetOp(astNode, 0)`.
+- `ScopeLeave` is a **no-op**: parse-time notes must survive until the
+  end-of-unit walk.
+
+Result: `t_lower3` (function + VAR-param procedure + calls + RETURN)
+**byte-matches**; suite **183/183**.
+
+## New finding — nested-array record field miscompiles
+
+Repro `docs/wip/nested-array-field-bug.mod`:
+
+```modula2
+TYPE PE = RECORD
+  name  : ARRAY [0..63] OF CHAR;
+  pname : ARRAY [0..3] OF ARRAY [0..63] OF CHAR;   (* nested *)
+END;
+VAR procs : ARRAY [0..3] OF PE;
+...
+procs[nProcs-1].pname[j][i] := name[i];
+```
 
-Then: `run_tests.sh` already needs `lower_ok t_lower3.mod`; expect
-**183/183** and **FIXPOINT OK**.
+`./M2` compiles it, but the **generated program segfaults** (rc 139;
+want 42).  A **flat** `pname : ARRAY [0..2047] OF CHAR` (manual stride)
+works.  This is a real user-facing bug — likely the element stride /
+descriptor-pointer setup for a nested-array field (same family as the
+Stage-A array-of-record stride fix in `QbeGen.ElemSize`, and
+`docs/wip/local-array-bug.mod`).
+
+It was hit directly: the V3-built compiler (`m2s2`) crashed on **every**
+input when `Lower`'s formal-name table used a nested-array field.
+Flattening fixed that crash.
+
+## Why self-host still fails
+
+After flattening, `m2s2` works on small programs (`t_exit`, a `CONST`
+program) but **still crashes (SIGABRT) on the full compiler input**
+(empty output).  Shrinking the tables (`MaxSym` 2048→512, `MaxProc`
+512→64) did **not** help, so it is not table size — it is a
+**size/complexity-triggered latent V3 codegen bug** in the compiled
+`Lower`/`M2P` code.  HEAD self-hosts; the ~250 added lines tip it over.
+
+This is the third instance of the same compiler fragility (local arrays,
+nested-array fields, and now this).  Finding the exact construct needs a
+bisect of the added code with a V3-built compiler — a dedicated task.
+
+## Next steps
+
+1. **Fix the nested-array-field bug in `QbeGen`** (like the Stage-A
+   stride fix): make `arr[i].field[j][k]` correct.  Add the repro to the
+   suite.  This is a real, self-contained bug and may also matter for the
+   size-triggered failure.
+2. **Bisect the size-triggered self-host crash**: build `m2s2`, then
+   compile subsets of the added `Lower`/`M2P` code with it to find the
+   construct that crashes.  Candidates: the flat-stride indexing, the
+   `CopySpan` offsets, or a specific grammar hook.
+3. Once self-host is green: keep `lower_ok t_lower3`, expect 183/183 +
+   FIXPOINT OK.
 
 ## Do NOT
 
-- Do not add `FORWARD` (breaks gm2 whole-program pass).
-- Do not edit `SymTab`.
-- Do not run legacy + `L` emitters concurrently (shared `QbeGen` state).
-- Do not leave an `ASTDUMP` in `M2.atg` (it was committed once by
-  accident and cost time — `git show v3-lower3-wip:compiler/src/M2.atg`
-  had it).
-
-## Also re-apply (two real bugs, from earlier this session)
-
-- `NkProcDecl` `op` must be `0` for a normal body (`CanProc` checks it;
-  default is `-1`).
-- `NkCall` child 0 is an `NkDesignator`, not a bare `NkIdent` — unwrap it
-  (`CalleeName`), else calls mangle as `$_0`.
+- No `FORWARD` (breaks gm2 whole-program pass).
+- No `SymTab` edits.
+- No arrays of records with aggregate (nested-array) fields in `Lower`.
+- No committed `ASTDUMP`.
 
 ## Resume
 

+ 61 - 0
docs/wip/nested-array-field-bug.mod

@@ -0,0 +1,61 @@
+MODULE NestedArrayFieldBug;
+(* Repro: a RECORD field that is a nested ARRAY OF ARRAY OF CHAR, accessed
+   as `arr[i].field[j][k]`, is miscompiled by V3 -> segfault (rc 139).
+   Observed 2026-10-05; the flat-array form works (rc 42).
+
+   Found while adding Lower's procedure-formal name table (a module-level
+   ARRAY OF RECORD with a `pname : ARRAY [0..31] OF ARRAY [0..63] OF
+   CHAR` field): the V3-built compiler crashed (SIGABRT) on every input.
+   Flattening the field to `ARRAY [0..2047] OF CHAR` fixed the crash.
+
+   Compare with docs/wip/local-array-bug.mod (local arrays) and the
+   Stage-A array-of-record stride fix (QbeGen.ElemSize).  Likely the
+   element stride / descriptor-pointer setup for a nested-array field is
+   wrong.  Expected exit 42; observed 139. *)
+
+VAR ExitCode : INTEGER;
+
+CONST MaxName = 63; MaxProc = 4;
+
+TYPE PE = RECORD
+  name  : ARRAY [0..MaxName] OF CHAR;
+  npar  : CARDINAL;
+  pname : ARRAY [0..3] OF ARRAY [0..MaxName] OF CHAR;  (* nested *)
+END;
+
+VAR procs : ARRAY [0..MaxProc-1] OF PE;
+    nProcs : CARDINAL;
+
+PROCEDURE Reset;
+BEGIN nProcs := 0 END Reset;
+
+PROCEDURE NoteProc (name : ARRAY OF CHAR);
+VAR i : CARDINAL;
+BEGIN
+  IF nProcs >= MaxProc THEN RETURN END;
+  i := 0;
+  WHILE (i < MaxName) AND (name[i] # CHR(0)) DO
+    procs[nProcs].name[i] := name[i]; INC(i)
+  END;
+  procs[nProcs].name[i] := CHR(0);
+  procs[nProcs].npar := 0; INC(nProcs)
+END NoteProc;
+
+PROCEDURE NoteParam (name : ARRAY OF CHAR);
+VAR j, i : CARDINAL;
+BEGIN
+  IF nProcs = 0 THEN RETURN END;
+  j := procs[nProcs-1].npar;
+  i := 0;
+  WHILE (i < MaxName) AND (name[i] # CHR(0)) DO
+    procs[nProcs-1].pname[j][i] := name[i]; INC(i)
+  END;
+  procs[nProcs-1].pname[j][i] := CHR(0);
+  procs[nProcs-1].npar := j + 1
+END NoteParam;
+
+BEGIN
+  Reset; NoteProc("Add"); NoteParam("a");
+  IF (nProcs = 1) AND (procs[0].pname[0][0] = "a")
+  THEN ExitCode := 42 ELSE ExitCode := 1 END
+END NestedArrayFieldBug.