|
|
@@ -1,203 +1,86 @@
|
|
|
-# m2compiler-V3 — status (2026-10-08)
|
|
|
+# m2compiler-V3 — status (2026-10-16)
|
|
|
|
|
|
-This answers "where are we after all the refactoring?".
|
|
|
+This answers "where are we now?".
|
|
|
|
|
|
-## MILESTONE (2026-10-08): L7 complete
|
|
|
+## Numbers
|
|
|
|
|
|
-**Lower is the only emitter**; the grammar's inline emit is gone from
|
|
|
-every production where it could be removed cleanly, `-lower` is retired,
|
|
|
-and the dead scaffolding is swept. Verified:
|
|
|
+- Suite **219/219** (`compiler/run_tests.sh`).
|
|
|
+- Self-hosting **FIXPOINT OK** — stage2 and stage3 images are
|
|
|
+ byte-identical (3,714,376 bytes).
|
|
|
+- Corpus **372 / 612** sources compile with 0 errors, **0 crashes**
|
|
|
+ (`tools/v3-corpus/corpus.sh` over the gm2 testsuite); the rest are
|
|
|
+ genuine language gaps or unresolved imports.
|
|
|
+- Branch **`resolve-check`**; tree clean.
|
|
|
|
|
|
-- `run_tests.sh` is **184/184**.
|
|
|
-- `bootstrap/fixpoint.sh` is **FIXPOINT OK** (3,392,601 bytes).
|
|
|
+## Architecture (current)
|
|
|
|
|
|
-The L7 flip itself (2026-10-07) was a no-op because `Lower` had been
|
|
|
-verified byte-identical to the legacy inline emit on every single-file
|
|
|
-test and every session. Since then:
|
|
|
-
|
|
|
-- the inline emit was deleted from statements, control flow,
|
|
|
- declarations, expressions/designators, calls and `ProcHeading`;
|
|
|
-- a latent VAR-actual **note-ring bug** was found and fixed
|
|
|
- (`RestoreSession` now clears the ring);
|
|
|
-- dead code was swept (tree-wide scan: 0 unreferenced procs);
|
|
|
-- `-lower` and the now-degenerate byte-compare tests were retired;
|
|
|
-- the **array-of-record inline length-word** bug was fixed (inline
|
|
|
- expansion for small arrays, a generated run-time init for large ones);
|
|
|
-- **statement-context result suffixes** (`F()^.x := e`) were added;
|
|
|
-- **composite function results** (record/array/set, via sret) were added,
|
|
|
- which also required storing a method's block declarations (fixing
|
|
|
- methods with local variables);
|
|
|
-- **class results** and **chained post-call method calls** (`F().M()`)
|
|
|
- were added;
|
|
|
-- **`GOTO`/labels** were added (the `Design` production was split into
|
|
|
- `Design` + `DesignTail` to resolve the LL(1) label/designator clash).
|
|
|
-
|
|
|
-## TL;DR
|
|
|
-
|
|
|
-- **The compiler works and self-hosts today.** `run_tests.sh` is
|
|
|
- **184/184** and `bootstrap/fixpoint.sh` is **FIXPOINT OK**
|
|
|
- (3,392,601 bytes, byte-identical stage2/stage3).
|
|
|
-- The **two-phase refactor (L0–L7) is done**: `Lower` emits the whole
|
|
|
- session from the AST and is the only active emitter.
|
|
|
-- What remains is cleanup (delete the last suppressed writers +
|
|
|
- `SaveSession`/`SetNoEmit`), language coverage (`GOTO`, generics,
|
|
|
- chained post-call calls), and robustness — see "What remains" below.
|
|
|
-
|
|
|
-## What the refactor actually is
|
|
|
-
|
|
|
-There are **two ways** the compiler can turn Modula-2 into QBE IR:
|
|
|
-
|
|
|
-1. **Legacy "inline" emit** — the grammar emits QBE IR *while* it
|
|
|
- parses. This used to be the default; it now only runs suppressed
|
|
|
- (its output is discarded) and is being deleted.
|
|
|
-2. **Lower (two-phase)** — the grammar builds an **AST**, and a
|
|
|
- separate module `Lower` walks the AST and emits QBE IR. This is
|
|
|
- now the **only active emitter** ("the flip").
|
|
|
-
|
|
|
-The refactor is being done in slices:
|
|
|
-
|
|
|
-| phase | what | state |
|
|
|
-|---|---|---|
|
|
|
-| L0–L2 | scalars, control flow (`IF`/`WHILE`/`FOR`/…) | done |
|
|
|
-| L3 | procedures, calls, the self-host root-cause fix | done |
|
|
|
-| L4 | builtins, `CASE`, `WITH` | done |
|
|
|
-| L5 | records/arrays/pointers/sets/strings, literals | done |
|
|
|
-| L6 | nested modules, classes, qualified names | done |
|
|
|
-| **L7** | **the flip: Lower emits the whole session** | **done** (flip landed 2026-10-07) |
|
|
|
-
|
|
|
-Everything is committed in small steps; the many commits are
|
|
|
-**incremental fixes to the new `Lower` path**, not changes to the
|
|
|
-working legacy compiler.
|
|
|
-
|
|
|
-## Where exactly we are (L7)
|
|
|
-
|
|
|
-The flip is driven per unit: the driver suppresses the grammar's
|
|
|
-inline emit (`SetNoEmit`), snapshots the QbeGen state before each
|
|
|
-unit's parse, restores it, and lets `Lower` emit the unit from the
|
|
|
-AST. This is now the *only* path — plain `./M2` runs it, and the
|
|
|
-self-hosting fixpoint runs it too. `EndSession` writes the single
|
|
|
-session image.
|
|
|
-
|
|
|
-The `-lower` flag survives only for the test harness: it names the
|
|
|
-image `<mod>L.ssa` instead of `<mod>.ssa` so `run_tests.sh` can still
|
|
|
-byte-compare two runs. Both runs now use Lower, so the comparison is
|
|
|
-degenerate; it will be retired together with the inline emit.
|
|
|
-
|
|
|
-Because `Lower` was verified byte-identical to the legacy inline emit
|
|
|
-on every single-file test and every session, flipping the default did
|
|
|
-not change any emitted image.
|
|
|
-
|
|
|
-## What L7 fixed so far
|
|
|
+```
|
|
|
+per unit:
|
|
|
+ Parse -> AST + declarations; each procedure body is skipped and
|
|
|
+ re-parsed at unit end, so forward references bind normally
|
|
|
+ Lower -> walk the AST, emit QBE (the only emitter)
|
|
|
+ Emit -> buffered image, EndModule
|
|
|
+```
|
|
|
|
|
|
-- `Lower` is **V3-legal** (forward references declared in `Lower.def`;
|
|
|
- V3 has no whole-module pass) and the `SymTab.FormalOk` open-array
|
|
|
- def/impl bug.
|
|
|
-- Whole-session tables + an open-addressing **hash index** for
|
|
|
- `FindSym`/`FindProc`.
|
|
|
-- **Symbol scoping**: a procedure's locals no longer collide with
|
|
|
- another procedure's (`HIGH(s)` picked the wrong `s`).
|
|
|
-- **Chunked sequences** where a list can exceed `AST.MaxChild`:
|
|
|
- `CASE` arms, multi-name `VAR` declarations, `CASE` bodies.
|
|
|
-- External procedures, indirect calls (proc-typed variables), `VAL`,
|
|
|
- `LEN`/`HIGH`, qualified calls, qualified proc-typed variables,
|
|
|
- const values, 1-char strings passed as `ARRAY OF CHAR`.
|
|
|
-- Per-unit global materialisation (`MaterializeReplace`).
|
|
|
-- Duplicate nested procedure names resolve by scope and uid.
|
|
|
+- **Two-phase parse.** Procedure bodies are deferred (module level) and
|
|
|
+ re-parsed after the unit's declarations, which removed the entire
|
|
|
+ forward-reference mechanism (`FwdVar*`, `Design`'s forward branch).
|
|
|
+ A module body may now call a procedure declared later; a nested-local
|
|
|
+ forward is a plain `201`. `-notwophase` restores the legacy inline
|
|
|
+ parse for debugging.
|
|
|
+- **No resolve/check pass.** The `Check` module was deleted once it was
|
|
|
+ proven vestigial (gutting it left the emitted `.ssa` byte-identical):
|
|
|
+ binding is late by construction.
|
|
|
+- **`Lower` is the only emitter**; the AST is the interface.
|
|
|
+
|
|
|
+## Language coverage
|
|
|
+
|
|
|
+Program modules; `CONST`/`VAR`/`TYPE` (scalars, subranges, enums,
|
|
|
+arrays, records incl. variants, sets, pointers); `WITH`/`IN`; whole
|
|
|
+array/record copy; `NEW`/`DISPOSE`; full expressions and control flow;
|
|
|
+`GOTO`/labels; procedures (value/`VAR` params, functions, recursion,
|
|
|
+`FORWARD`, nested procedures, procedure values); Clarion-form `CLASS`
|
|
|
+(methods, single inheritance, virtual dispatch); PIM coroutines;
|
|
|
+Unicode (`UCHAR`/`UString`, UTF-8 edges); ISO type conversions `T(x)`;
|
|
|
+separate compilation (`FROM`/`IMPORT`, qualified `L.x`, opaque types);
|
|
|
+**generics** (ISO 10514-2 refinement): type/value/procedure
|
|
|
+parameters, constraints, nested generics, generic classes.
|
|
|
+
|
|
|
+## Landed recently (this thread)
|
|
|
+
|
|
|
+- AST + `Lower` (L0–L7) — `Lower` emits the whole session from the AST.
|
|
|
+- Two-phase parse; forward machinery deleted; `Check` deleted.
|
|
|
+- Generics increment 3: value/procedure parameters, constraint model,
|
|
|
+ nested generics, generic classes via generic modules.
|
|
|
+- ISO type conversions `T(x)` (incl. CONST folding): corpus 365 → 372.
|
|
|
+- Clean `M2` exit codes (0 ok / 1 errors / 2 usage-IO).
|
|
|
+- Bug fixes: `PopPos` wiping repass diagnostics; global record data
|
|
|
+ size (`w 0` → `l 0` for pointer/`LONGINT`/proc fields); template
|
|
|
+ arithmetic over a type formal; proc-typed formals in DEF/IMPL;
|
|
|
+ imported procedures used as values.
|
|
|
+- Showcase `showcase23` (generic doubly linked list).
|
|
|
|
|
|
## What remains
|
|
|
|
|
|
-Cleanup with no codegen gaps left. **Not** purely mechanical: many
|
|
|
-`QbeGen.*` calls are semantic queries or parser state, not emit, and
|
|
|
-must stay — `TopLoop`/`PushLoop`/`PopLoop` (validates `EXIT`),
|
|
|
-`TopWith`/`PushWith`/`PopWith` (validates a bare field name in `WITH`),
|
|
|
-`IsImm`/`Fold2`/`NormInt`/`NormReal`/`StrFold`/… (constant values that
|
|
|
-`Lower` consumes via `NoteConstVal`), `LocFull`, `Mangled`, `ConstInt`.
|
|
|
-Only the true emission calls are removed, production group by group,
|
|
|
-with suite + fixpoint after each.
|
|
|
-
|
|
|
-Done:
|
|
|
-1. Builtin statements (`EXIT`, `INCL/EXCL`, `INC/DEC`, `NEW`, `DISPOSE`).
|
|
|
-2. Control flow (`IF`/`WHILE`/`REPEAT`/`LOOP`/`FOR`/`CASE`), `RETURN`,
|
|
|
- `HALT`.
|
|
|
-3. Declarations/lifecycle: `DeclVar`/`DeclConst`, the module/impl/
|
|
|
- nested/class `BeginInit`/`EndInit`/`BeginBody`/`EndModule`,
|
|
|
- `FlushPend`, `FwdPatch`. (This exposed and fixed a real bug — see
|
|
|
- below.)
|
|
|
-
|
|
|
-Still to do:
|
|
|
-4. **Expressions/designators/calls: DONE (2026-10-08).** The runtime
|
|
|
- emit is gone from `SimExpr`, `Term`, `Expr`, `Fact`, `Design`,
|
|
|
- `ResultComp`, `ArgList`, `ActParam` and assignment; a runtime result
|
|
|
- is the non-constant sentinel `"@"` (which `IsImm`/`ConstInt`
|
|
|
- reject), while constant folding (`NegFold`/`Fold2`/`StrFold`) and
|
|
|
- every check stay.
|
|
|
- The coupled change that had blocked this landed together: `Fact`
|
|
|
- drops the `NoteAddr` chain, `ActParam` checks VAR-actual
|
|
|
- addressability by AST kind (`NkDesignator`, or `NkStrLit`) instead
|
|
|
- of the `NoteAddr`/`AddrOfVal` chain and drops `CallArg`, and
|
|
|
- `ArgList` drops `CallBegin`/`CallEnd`. (`NoteAddr`/`AddrOfVal`
|
|
|
- remain, now used only by `Lower` for its own emit.)
|
|
|
-5. **Proc headings, set/ctor literals, string pooling: remaining.**
|
|
|
- `ProcHeading` still calls `BeginFunc`/`FuncParam`/`EndFuncHeader`/
|
|
|
- `EndFunc`/`Mangled`/`SetFuncRes`/`AbortFunc`; note `FuncParam` is a
|
|
|
- check that relies on `BeginFunc` resetting `nPar`, so these must be
|
|
|
- split rather than dropped. `SetLit`/`TypedBraceLit`
|
|
|
- (`NewSetTemp`/`SetZero`/`SetRange`/`SetBit`/`Ctor*`) and the
|
|
|
- `DeclStr`/`DeclUStr` string pool are value-ish; `CopyOp`/`IsImm`/
|
|
|
- `IntStr`/`Fold2`/`Norm*`/`Conv*` are the value layer and stay.
|
|
|
-6. Retire the `-lower` test flag and the now-degenerate byte-compare
|
|
|
- tests. **Done 2026-10-08:** `-lower`/`IsLowerParam`/`lowerTest` and
|
|
|
- `EndSession`'s `testSuffix` are gone, and the 23 `lower_ok` cases
|
|
|
- dropped (suite now **181**; the programs are still covered by the
|
|
|
- `expect_run`/session cases).
|
|
|
-7. Remove the dead scaffolding. Done 2026-10-08: `LowerSession`,
|
|
|
- `LowerUnit`, the whole `Can*` capability-walker family,
|
|
|
- `CanLower`/`CanExpr`/`CanDecls`/`CanProc`, the dead
|
|
|
- `SetLower`/`lowerRequested`, plus every compiler proc a tree-wide
|
|
|
- scan (tests included) finds unreferenced — `QbeGen`'s
|
|
|
- `CloseModule`/`FwdPatch`/`FwdElemPatch`/`Remark`/`StrLen`,
|
|
|
- `SymTab`'s `CurModule`/`CurUnit`/`ProcDepth`/`ParamName`/
|
|
|
- `ModDefined`/`ModImplemented`/`Qual*`/`NestArray`/`NewSub`/
|
|
|
- `SetElemCheck`/`SetFor`/`UnaryCheck`/`FwdVarKind`, and `AST`'s
|
|
|
- `TxtLen`/`SetKind` (`AST.AddChild` stays; `tests/t_ast.mod` uses it).
|
|
|
- Still needed until the inline emit is gone:
|
|
|
- `SaveSession`/`RestoreSession`/`SetNoEmit` (the grammar still emits
|
|
|
- suppressed) and `MaterializeGlobals`/`MaterializeReplace` (called
|
|
|
- from `LowerOneUnit`).
|
|
|
-
|
|
|
-Note: full deletion to zero is gated on moving constant folding into
|
|
|
-`Lower` (item 4). Until then the expression/value layer stays.
|
|
|
-
|
|
|
-### Fixed (2026-10-08): the note ring leaked the grammar's notes into Lower
|
|
|
-
|
|
|
-Removing `QbeGen.DeclVar` from `VarDecl` had made `Lower` resolve one
|
|
|
-VAR actual to the wrong symbol (`Showcase11`: `Strings.Replace`'s `src`
|
|
|
-became `$Strings_nl`). Root cause: `QbeGen.AddrOfVal` scans the whole
|
|
|
-256-entry `nVal`/`nAddr` VAR-actual note ring, and `RestoreSession`
|
|
|
-restored `nn` but never cleared the ring — so the grammar's suppressed
|
|
|
-parse-time `NoteAddr` entries survived into `Lower`. Their keys are
|
|
|
-temps that restart per function, so a stale entry could false-match
|
|
|
-`Lower`'s own key. Removing `DeclVar` (fewer `AllocLocal`/`NewTemp`
|
|
|
-calls) shifted those keys and exposed it. `RestoreSession` now clears
|
|
|
-the ring; `Lower` re-notes its own VAR actuals, so an empty ring is
|
|
|
-correct. The fix is output-neutral on `Showcase11` and corrects some
|
|
|
-latent resolutions in the compiler's own image (fixpoint size moved).
|
|
|
-
|
|
|
-
|
|
|
-## Are we near completion?
|
|
|
-
|
|
|
-- **Of the working compiler:** yes — it is done, green, and
|
|
|
- self-hosting, and it now emits through `Lower`.
|
|
|
-- **Of the L7 flip:** the flip is done; only the mechanical deletion of
|
|
|
- the unused inline emitter remains. It is not required for the
|
|
|
- compiler to function — it is a quality/architecture goal.
|
|
|
+1. **Corpus language gaps** — top `LANG` buckets are `not supported
|
|
|
+ yet`, `undeclared identifier`, `invalid call`; separate genuine gaps
|
|
|
+ from missing stdlib imports (`LIB`), then fix the top bounded one.
|
|
|
+ See `docs/analysis_gaps.md`.
|
|
|
+2. **The size-sensitive V3 miscompile** — the long-standing "removing
|
|
|
+ `ASTDUMP` from `M2.atg` breaks the fixpoint" has never been
|
|
|
+ root-caused. A real correctness landmine.
|
|
|
+3. **Robustness** — fixed table limits resolve silently when exceeded;
|
|
|
+ weak error recovery; error-message polish.
|
|
|
+4. **Library holes** — channel/device framework (`docs/wip/`).
|
|
|
+5. **Housekeeping** — consolidate `docs/summary_*.md` (see
|
|
|
+ `docs/INDEX.md`); decide on merging/pushing `resolve-check`.
|
|
|
|
|
|
## How to check for yourself
|
|
|
|
|
|
```sh
|
|
|
cd compiler
|
|
|
-./build.sh # rebuild
|
|
|
-./run_tests.sh # 184/184
|
|
|
+./build.sh # M2.atg -> scanner/parser/driver -> ./M2
|
|
|
+./run_tests.sh # 219/219
|
|
|
cd ..
|
|
|
-bootstrap/fixpoint.sh # FIXPOINT OK
|
|
|
+./bootstrap/fixpoint.sh # FIXPOINT OK (byte-identical stage2/stage3)
|
|
|
```
|
