Jelajahi Sumber

docs: add a plain-language status of the compiler and the L7 flip

Explains the two code paths (legacy inline emit vs Lower), what is
done, where the flip is (736/736 functions, ~74% byte-identical, first
divergence at the generic ArgList call), what remains, and how close
completion is.
Eric Streit 5 hari lalu
induk
melakukan
e12195bbc6
1 mengubah file dengan 115 tambahan dan 0 penghapusan
  1. 115 0
      docs/STATUS.md

+ 115 - 0
docs/STATUS.md

@@ -0,0 +1,115 @@
+# m2compiler-V3 — status (2026-10-06)
+
+This answers "where are we after all the refactoring?".
+
+## TL;DR
+
+- **The compiler works and self-hosts today.**  `run_tests.sh` is
+  **204/204** and `bootstrap/fixpoint.sh` is **FIXPOINT OK**
+  (3,758,751 bytes, byte-identical stage2/stage3).
+- The **two-phase refactor (L0–L6) is done and committed**.  The last
+  step, **L7 "the flip"**, is **in progress** and is **opt-in**
+  (`-lower`); it does **not** affect the normal build.
+- The flip already produces the full compiler image (736/736
+  functions), but it is only **~74 % byte-identical** to the legacy
+  image: the first difference is at line 89,349 of 120,266.
+- **We are not near completion of the flip.**  The mechanism is done;
+  what remains is closing the last code-generation gaps until the
+  images are byte-identical.  Each gap found so far has been a small,
+  local fix, but the session is huge and byte-identity is exacting.
+
+## What the refactor actually is
+
+There are **two ways** the compiler can turn Modula-2 into QBE IR:
+
+1. **Legacy "inline" emit** — the grammar emits QBE IR *while* it
+   parses.  This is the default and is what the working compiler and
+   the self-hosting fixpoint use.
+2. **Lower (two-phase)** — the grammar builds an **AST**, and a
+   separate module `Lower` walks the AST and emits QBE IR.  This is
+   selected with the **`-lower`** flag and is the goal of the
+   refactor ("the flip" = make Lower the only emitter and delete the
+   inline emit).
+
+The refactor is being done in slices:
+
+| phase | what | state |
+|---|---|---|
+| L0–L2 | scalars, control flow (`IF`/`WHILE`/`FOR`/…) | done |
+| L3 | procedures, calls, the self-host root-cause fix | done |
+| L4 | builtins, `CASE`, `WITH` | done |
+| L5 | records/arrays/pointers/sets/strings, literals | done |
+| L6 | nested modules, classes, qualified names | done |
+| **L7** | **the flip: Lower emits the whole session** | **in progress** |
+
+Everything is committed in small steps; the many commits are
+**incremental fixes to the new `Lower` path**, not changes to the
+working legacy compiler.
+
+## Where exactly we are (L7)
+
+The flip is driven per unit: the driver suppresses the grammar's
+inline emit (`SetNoEmit`), snapshots the QbeGen state before each
+unit's parse, restores it, and lets `Lower` re-emit the unit.  On the
+compiler's own sources:
+
+- `-lower` emits the **full** image: **736 functions** (same as
+  legacy), ~1.5 s.
+- It is **~74 % byte-identical** before the first divergence:
+  `first diff at byte 2,935,735, line 89,349` of `120,266`.
+- Progress of the first divergence across the session:
+  `line 9 → 327 → 1124 → 3437 → 6961 → 56294 → 70934 → 82194 → 86942 → 89349`.
+
+The first divergence is now the generic grammar `ArgList` call, which
+has **9 arguments** but `AST.MaxChild = 8` caps a call at 7, so `Lower`
+drops the tail.
+
+## What L7 fixed so far
+
+- `Lower` is **V3-legal** (forward references declared in `Lower.def`;
+  V3 has no whole-module pass) and the `SymTab.FormalOk` open-array
+  def/impl bug.
+- Whole-session tables + an open-addressing **hash index** for
+  `FindSym`/`FindProc`.
+- **Symbol scoping**: a procedure's locals no longer collide with
+  another procedure's (`HIGH(s)` picked the wrong `s`).
+- **Chunked sequences** where a list can exceed `AST.MaxChild`:
+  `CASE` arms, multi-name `VAR` declarations, `CASE` bodies.
+- External procedures, indirect calls (proc-typed variables), `VAL`,
+  `LEN`/`HIGH`, qualified calls, qualified proc-typed variables,
+  const values, 1-char strings passed as `ARRAY OF CHAR`.
+- Per-unit global materialisation (`MaterializeReplace`).
+- Duplicate nested procedure names resolve by scope and uid.
+
+## What remains
+
+1. **Calls with more than 7 arguments** (the `ArgList` case):
+   chunk the call actuals and make the argument loops follow the
+   `NkBlock` chunks.  (A prototype regressed `AstCallNode`'s legacy
+   emit and was reverted.)
+2. Whatever the next first-divergence reveals.  Historically each fix
+   has exposed one more small gap; there may be a few more.
+3. Only then: delete the grammar's inline emit (the literal flip) and
+   remove the `-lower` flag.
+
+## Are we near completion?
+
+- **Of the working compiler:** yes — it is done, green, and
+  self-hosting.  The refactor has not broken it.
+- **Of the L7 flip:** the *hard architectural part is finished*
+  (per-unit emit, scoping, tables, the AST chunking mechanism).  The
+  remainder is a **tail of codegen gaps** that only matter because the
+  verification is exact byte-identity over a 120k-line session.
+  Realistically a handful more small fixes; not a rewrite.  It is not
+  required for the compiler to function — it is a quality/architecture
+  goal.
+
+## How to check for yourself
+
+```sh
+cd compiler
+./build.sh          # rebuild
+./run_tests.sh      # 204/204
+cd ..
+bootstrap/fixpoint.sh   # FIXPOINT OK
+```