Browse Source

docs: refresh README, compiler notes and add tutorial

Eric Streit 1 day ago
parent
commit
a8cd5f9bce
5 changed files with 335 additions and 51 deletions
  1. 4 2
      CALC-COMPILE.md
  2. 45 30
      README.md
  3. 5 2
      SUMMARY.md
  4. 33 17
      TP3-COMPILER.md
  5. 248 0
      TUTORIAL.md

+ 4 - 2
CALC-COMPILE.md

@@ -14,7 +14,7 @@ The measurements were taken against the tree at tag `v-TP3-INTERRUPT-RESTORE`
 tagged `v-TP3-CALC-LAYOUT`, and §1's rung table and the "Session state" section
 are the current figures).
 
-## Current status (2026-10-27)
+## Current status (2026-10-28)
 
 The goal of this document is met: `CALC.PAS` compiles to its final `.` and the
 smoke test boots the resulting `.COM`.  The stage plan below is a historical
@@ -23,7 +23,9 @@ state when that stage was written, not the current compiler.
 
 Current measurements: **27,519 B code, 19,694 B data, 38,497 B `.COM`**;
 runtime **10,975 bytes / 82 entries**.  The surviving `pascsrc` TP3 corpus is
-**71 / 71 OK** and is checked by `tests/corpus_check.py`.
+**71 / 71 OK** and is checked by `tests/corpus_check.py`.  The compiler is now
+split into focused modules; see `SUMMARY.md` and `TUTORIAL.md` for the current
+layout and build walkthrough.
 
 What is still outside the automated CALC evidence: a full Help/heap session and
 the complete editor/menu workflow.  The smoke test covers the welcome screen,

+ 45 - 30
README.md

@@ -8,31 +8,29 @@ and behaviours are reconstructed from the disassembled TP3.0 source
 ## State
 
 **Where things are: `SUMMARY.md`.** It holds the milestone table, the list of
-checks and what each one cannot see, and the numbers behind every claim.
-
-There used to be a milestone table here as well, and it drifted six milestones
-behind while still reading as authoritative: it said "no compiled program has
-been executed yet" and "`CmdRun` is a stub and the runtime blob does not exist"
-through five tagged milestones in which those three things all became true. A
-second copy of a status table is a claim about the world that expires silently
-and is never noticed expiring, so there is one now, and this file says where it
-is.
-
-In short, current as of `v-TP3-LANGUP`:
-
-- The shell and the WordStar-style editor are done.
-- The compiler front end and 8086 code generator are done for the language
-  subset the fixtures cover, and the emitted `.COM` images **run** under
-  `qemu-system-i386` and Exec86.
-- **CALC.PAS compiles and runs.** The generated CALC.COM reaches the welcome
-  screen, grid and command menu, accepts a numeric formula or text cell, and
-  exits cleanly through `/Q`.
-- **The shell's `R` key is still a stub.** There is no in-process 8086
-  interpreter, so nothing runs a `.COM` from inside `tpshell` itself; execution
-  happens under the test harness or the CALC launcher below.
-- The next capacity limit is CALC's full Help/heap use (16,281-byte `Pages`
-  plus a 16 KiB stack); the current flat-segment layout has less headroom than
-  that. `SUMMARY.md` and `CALC-COMPILE.md` record the details.
+checks and what each one cannot see, and the numbers behind every claim. The
+short version:
+
+- The shell, WordStar-style editor, compiler, linker and in-process 8086
+  interpreter all build and run.
+- The compiler is split into focused modules: `CompilerShared`,
+  `CompilerUtil`, `CompilerSource`, `CompilerSymbols`, `CompilerLex`,
+  `CompilerExpr`, `CompilerIO`, `CompilerEmit` and the statement/declaration
+  driver in `Compiler.mod`.
+- The surviving TP3 corpus compiles **71/71**; `CALC.PAS` compiles to its final
+  `.COM` and runs through the welcome screen, grid, menu and a formula.
+- Emitted `.COM` images run under both `qemu-system-i386` and Exec86; the
+  shell's **`R` key now runs the linked image in-process**.
+- Current measured runtime: **10,975 bytes / 82 entries**. CALC:
+  **38,497-byte `.COM`**.
+- Latest code checkpoint: `v-TP3-EXPR-IO`; latest documentation checkpoint:
+  `v-TP3-DOCS`.
+
+The remaining work is TP3-conformance and verification, not basic compilation:
+compiler directives (`$K`, `$V`, and explicit handling of the rest), real
+DOS-filesystem execution for the file routines, the full CALC Help/heap path,
+the 16-argument call limit, and finishing the statement/declaration split.
+`SUMMARY.md` lists these as the next milestones in priority order.
 
 ## Build
 
@@ -48,7 +46,12 @@ make
 The shell runs in raw mode: it needs a real ANSI terminal, no wrappers.
 
 To run the checks: `cd shell && bash tests/run_all.sh`. It is not part of
-`make`, because it needs qemu and takes a few minutes.
+`make`, because it needs qemu and takes a few minutes. The mutation suite is
+`bash tests/nonvacuity.sh`.
+
+**New here? Start with [`TUTORIAL.md`](TUTORIAL.md).** It walks through the
+build, the shell, editing and compiling a program, running the emitted image,
+and the test harnesses.
 
 ## CALC launcher
 
@@ -84,8 +87,19 @@ under the FreeDOS 1.4 LiveCD.
 - `shell/Editor.def|mod` — WordStar-style full-screen editor.
 - `shell/Shell.mod` — TP3.0 main-menu dispatcher (`kmenu`/`kcmdtab`),
   work-file load/save with `.BAK` rotation, directory listing, options menu.
-- `shell/Compiler.def|mod` — lexer, parser, symbol table and the 8086 code
-  generator. `Compiler.CodeByteAt` exists so a test can read the emitted image.
+- `shell/Compiler.def|mod` — the compiler's public interface and the
+  statement/declaration/driver core.
+- `shell/CompilerShared.def` — shared compiler state, constants and types.
+- `shell/CompilerUtil.def|mod` — pure character/word/bit/string helpers.
+- `shell/CompilerSource.def|mod` — source cursor and `{$I}` expansion.
+- `shell/CompilerSymbols.def|mod` — symbol table, lookup, insertion and scope
+  hiding.
+- `shell/CompilerLex.def|mod` — keyword table, scanning, constants and the
+  string-literal pool.
+- `shell/CompilerExpr.def|mod` — expression parser and value construction.
+- `shell/CompilerIO.def|mod` — standard-procedure dispatch.
+- `shell/CompilerEmit.def|mod` — primitive 8086 emitters and patch machinery.
+  `Compiler.CodeByteAt` exists so a test can read the emitted image.
 - `shell/Runtime.def|mod` — the 8086 runtime library, built as a byte blob at
   run time and copied to the front of every image. Its entries are addressed by
   name from `Runtime.def`; the emitter procedures are audited name-against-decode.
@@ -97,7 +111,8 @@ under the FreeDOS 1.4 LiveCD.
   proves each one can go red. `tests/fixtures/expected.tsv` is the live
   fixture matrix and is asserted against, not copied into prose.
 - `SUMMARY.md` — **the state document**: milestones, checks, numbers, and what
-  is still missing.
+  is still missing. It also carries the prioritized next milestones.
+- `TUTORIAL.md` — a hands-on walkthrough from build to compile/run and tests.
 - `TP3-COMPILER.md` — how the compiler works, the build recipe, and the
   codegen bug classes with their fixes.
 - `TP3-EDITOR.md` — the editor's key handling and screen layout.
@@ -122,7 +137,7 @@ Exact TP3.0 screen: `Logged drive`, `Active directory`, `Work file`,
 | `M` | set main file |
 | `E` | edit work file |
 | `C` | compile (wired: TP3 error report + editor jumps to the error) |
-| `R` | run — **stub**, prints `Interpreter pending - compiled code is in memory` |
+| `R` | run the linked image in-process through Exec86 and report its exit status |
 | `S` | save with `^Z` EOF; old version → `.BAK` |
 | `D` | directory listing, DOS `*.*` mask, `k bytes free` |
 | `O` | options submenu (compile target, Code/Data/Stack, params) |

+ 5 - 2
SUMMARY.md

@@ -2101,8 +2101,11 @@ independently-scanned inventory at all.
 
 ## Reference material
 
-- `TP3-COMPILER.md` — the compiler: what was changed, the `Skip` bug class,
-  the `rel16` off-by-2, standard procedures, current matrix.
+- `README.md` — project overview, current state, build, layout and shell keys.
+- `TUTORIAL.md` — hands-on build → edit → compile → run → test walkthrough.
+- `TP3-COMPILER.md` — the compiler's implementation history: what was
+  changed, the `Skip` bug class, the `rel16` off-by-2, standard procedures and
+  the codegen bug classes.
 - `TP3-EDITOR.md` / `TP3-EDITOR-PSEUDOCODE.md` — the original TP3 editor,
   from TPSRC5/TPSRC6.
 - `RESUME-TP3.md` — book-derived reference for the 8086 code TP3 generates

+ 33 - 17
TP3-COMPILER.md

@@ -1,25 +1,41 @@
 # TP3-comp — Turbo Pascal 3 clone in GNU Modula-2 (ISO)
 
-**Status:** compiler-core (`Compiler.mod`) + shell compiled clean with
-`gm2 -fiso -Wall`, and the ISO shell `tpshell` builds and runs (ESC-key
-driven, TP3-style DOS screen redraws).  This is the working *implementation*
-companion — the human overview is `TP3-COMPILER.md` history summary and
-`RESUME-TP3.md`.
+**Status (2026-10-28):** the shell, editor, compiler, linker and in-process
+8086 interpreter build and run. The compiler is split into `CompilerShared`,
+`CompilerUtil`, `CompilerSource`, `CompilerSymbols`, `CompilerLex`,
+`CompilerExpr`, `CompilerIO`, `CompilerEmit` and the statement/declaration
+driver in `Compiler.mod`. The surviving TP3 corpus compiles **71/71**, CALC
+produces its final `.COM`, and the `.COM` execution, runtime-entry, frame,
+encoding, non-vacuity and CALC checks are green.
 
-## Build (proven, two-phase — the ONLY recipe that beats GNU gm2's
-pass-3 "too many errors / too many identifiers" cap on compiler-sized
-whole-program programs)
+The human overview and the current milestone/roadmap list are in
+`SUMMARY.md`; the hands-on build/edit/compile/run walkthrough is in
+`TUTORIAL.md`. The rest of this file is the compiler's implementation history
+and the codegen bug classes it records.
 
+## Build
+
+`make` is the authoritative recipe:
+
+```sh
+cd shell
+make
+./tpshell
 ```
-gm2 -fiso -c Compiler.mod          # phase A: small per-module pass
-gm2 -fiso -fgen-module-list=modules.lst -o /dev/null \
-    Compiler.mod Term.o TextBuf.o Posix.o Editor.o  # phase B1: build list
-gm2 -fiso -fuse-module-list=modules.lst -o tpshell \
-    Shell.mod Compiler.mod Term.o TextBuf.o Posix.o Editor.o  # phase B2: link
+
+It performs the required two-phase GNU Modula-2 link. A single whole-program
+`gm2 -o` pass trips the compiler's pass-3 identifier/error cap on a
+compiler-sized program. The generated object list includes the compiler
+modules in dependency order:
+
+```text
+CompilerShared.o CompilerUtil.o CompilerSource.o CompilerSymbols.o
+CompilerLex.o CompilerExpr.o CompilerIO.o CompilerEmit.o Compiler.o
+Runtime.o Linker.o Exec86.o
 ```
-(Equivalent make target `tpshell` in `shell/Makefile`.) The single `gm2 -o`
-whole-program pass 3 silently caps identifier/error counts — see
-`build_tpshell.sh`.)
+
+The diagnostic `ParseExpr: too many errors in pass 3` during phase B1 is the
+expected module-list generation rollup, not a real failure.
 
 ## What was changed inside `Compiler.mod` (all ISO-legal, gm2-verifiable)
 
@@ -311,7 +327,7 @@ reported `3,3,3,3` while every ordering comparison was inverted.
 Several harnesses live in `shell/tests/` -- in the repo on purpose, because
 `/tmp` is wiped between sessions and an earlier /tmp-only harness was lost
 with it.  `shell/tests/run_all.sh` runs them all; its header says which is
-which.  `shell/SUMMARY.md` (at the repo root) is the state-of-the-project
+which.  `SUMMARY.md` (at the repo root) is the state-of-the-project
 document.
 
 ### `run_compile_tests.sh` -- compiler front end, no pty, instant

+ 248 - 0
TUTORIAL.md

@@ -0,0 +1,248 @@
+# TP3-comp tutorial
+
+This is a hands-on path from a clean checkout to a running shell, a compiled
+Pascal program, and the test suite. It assumes the repository root is
+`TP3-comp/`.
+
+The short version:
+
+```sh
+cd TP3-comp/shell
+make
+./tpshell
+```
+
+Then press `W` to name a work file, `E` to edit it, `C` to compile, and `R`
+to run it. The rest of this page explains each step and the command-line
+alternatives.
+
+## 1. What you need
+
+- GNU Modula-2 (`gm2`, tested with 16.0.1)
+- a C compiler for the `DEFINITION MODULE FOR "C"` bridge
+- `qemu-system-i386` for the execution tests
+- `make`
+- a real ANSI terminal for the interactive shell
+
+The build itself only needs `gm2`, `cc` and `make`. qemu is needed by
+`tests/run_all.sh`, `tests/run_com_exec.py`, `tests/run_exec86.py` and
+`tests/calc_cli.py`.
+
+## 2. Build the shell
+
+From the repository root:
+
+```sh
+cd shell
+make
+```
+
+This builds `tpshell` (and `tp3cfg`, the terminal configurator). The
+authoritative recipe is `shell/Makefile`; it uses GNU Modula-2's two-phase
+link because a single whole-program `gm2 -o` pass trips the compiler's
+pass-3 limit on a compiler-sized program.
+
+A clean build should end with `tpshell` and no real compile errors. The one
+line that mentions `ParseExpr: too many errors in pass 3` is the expected
+phase-1 generation rollup and is tolerated by the recipe.
+
+## 3. Start the shell
+
+```sh
+./tpshell
+```
+
+The shell uses raw terminal mode and draws the TP3.0 main menu:
+
+- `L` — logged drive / directory
+- `A` — active directory
+- `W` — work file
+- `M` — main file
+- `E` — edit
+- `C` — compile
+- `R` — run
+- `S` — save
+- `D` — directory listing
+- `O` — options
+- `Q` — quit
+
+`R` runs the linked image in the in-process Exec86 interpreter. It is no
+longer a stub.
+
+## 4. Your first program
+
+Inside `tpshell`:
+
+1. Press `W` and enter `hello` (or `hello.pas`). The shell adds `.PAS` when
+   needed.
+2. Press `E` to open the editor.
+3. Enter:
+
+   ```pascal
+   program hello;
+   begin
+     writeln('Hello from TP3-comp')
+   end.
+   ```
+
+4. Press `^K-D` to return to the shell.
+5. Press `S` to save. The previous version, if any, becomes `.BAK`.
+6. Press `C` to compile.
+7. Press `R` to run the image.
+
+If compilation fails, the shell reports the TP3-style error number and jumps
+the editor to the error position. The menu's status line shows the current
+work file and free space.
+
+### Editor essentials
+
+- `^K-D` — leave the editor and return to the shell (then `S` saves)
+- `^S` / `^D` / `^E` / `^X` — move by character or line
+- `^Q-F` — search
+- `^Q-A` — replace
+- `^K-B` / `^K-K` — mark a block
+- `^K-C` / `^K-V` / `^K-Y` — copy / move / delete block
+
+The complete key list is in `README.md` and `TP3-EDITOR.md`.
+
+## 5. Compile without the interactive shell
+
+The fast compiler loop is `tests/CompileTest.mod`, built as `compiletest`.
+Build it once, then feed it paths (one per line):
+
+```sh
+cd shell
+tests/run_compile_tests.sh --build-only
+
+printf '%s\n' tests/fixtures/t01_minimal.pas | ./compiletest
+```
+
+You should see a line like:
+
+```text
+tests/fixtures/t01_minimal.pas  OK  code=34 data=5
+```
+
+`compiletest` loads the file exactly as the shell does (LF to CR, `^Z` ends
+the text), calls `Compiler.Compile`, and prints the error position and a
+caret on failure. It is much faster than driving the editor.
+
+To compile the whole fixture matrix:
+
+```sh
+tests/run_compile_tests.sh
+```
+
+The expected verdicts are in `tests/fixtures/expected.tsv`, including error
+numbers and positions.
+
+## 6. Run a compiled image
+
+The normal test path builds a floppy image, boots the `.COM` under
+`qemu-system-i386`, captures the serial output, and compares it byte for byte
+with a fixture `.out` file:
+
+```sh
+cd shell
+python3 tests/run_com_exec.py t01_minimal
+```
+
+Every executed fixture is also checked by the independent Exec86 interpreter:
+
+```sh
+python3 tests/run_exec86.py t01_minimal
+```
+
+For a larger, interactive example, build and boot CALC:
+
+```sh
+python3 tests/calc_cli.py
+python3 tests/calc_smoke.py
+```
+
+CALC needs its `CALC.INC` include file; `calc_cli.py` stages the source in a
+temporary directory for you. See `FREEDOS-CALC.md` for a manual FreeDOS/qemu
+setup.
+
+## 7. Run the whole check suite
+
+```sh
+cd shell
+bash tests/run_all.sh
+```
+
+This runs the compiler matrix, the shell/editor UI tests, frame and encoding
+checks, runtime entry tests, the `.COM` linker checks, the corpus census,
+CALC, and the execution harnesses.
+
+Then run the mutation suite:
+
+```sh
+bash tests/nonvacuity.sh
+```
+
+It deliberately breaks a check's subject and requires the check to go red.
+A green `nonvacuity.sh` is what proves that the other checks are not merely
+passing by accident.
+
+Current expected totals are kept in `SUMMARY.md`; do not copy them into every
+document, because they expire. At the `v-TP3-EXPR-IO` checkpoint they were:
+
+- compile matrix: **154/154**
+- non-vacuity: **121 ok, 0 failed**
+- corpus: **71/71**
+- runtime: **10,975 bytes / 82 entries**
+- CALC: **38,497-byte `.COM`**
+
+## 8. Where the compiler lives
+
+The compiler used to be one large `Compiler.mod`. It is now split into:
+
+| Module | Responsibility |
+|---|---|
+| `CompilerShared.def` | compiler-wide constants, state and types |
+| `CompilerUtil.def/.mod` | pure character, word, bit and string helpers |
+| `CompilerSource.def/.mod` | source cursor and `{$I}` include expansion |
+| `CompilerSymbols.def/.mod` | symbol table and scope lookup |
+| `CompilerLex.def/.mod` | keyword table, scanning, constants, string pool |
+| `CompilerExpr.def/.mod` | expression parser and value construction |
+| `CompilerIO.def/.mod` | standard-procedure dispatch |
+| `CompilerEmit.def/.mod` | primitive 8086 emitters and patch machinery |
+| `Compiler.def/.mod` | public API plus statements, declarations and driver |
+
+`Compiler.def` currently exposes a deliberately broad internal interface so
+the mutually recursive parser modules can call each other. Finishing the
+modularization means shrinking that interface and extracting the statement
+and declaration groups.
+
+## 9. Common problems
+
+**`./tpshell` draws nothing or behaves strangely.** It needs a real ANSI
+terminal in raw mode. Run it directly in a terminal, not through a wrapper
+that strips escape sequences.
+
+**`make` reports `too many errors in pass 3`.** This is expected for the
+phase-1 module-list generation. The actual link is the following
+`-fuse-list` command in `shell/Makefile`.
+
+**`compiletest` is missing.** It is a test harness, not part of `make`.
+Run `tests/run_compile_tests.sh --build-only`.
+
+**Execution tests cannot find qemu.** Install `qemu-system-i386`; the tests
+invoke it by path and will report the missing binary.
+
+**A `.COM` test says “comtest wrote no .COM”.** Build the harness first with
+`tests/run_com_tests.sh`, or let `run_all.sh` build it.
+
+## 10. What to work on next
+
+The corpus target is met. The remaining work is TP3 conformance and proof:
+
+1. `v-TP3-DIRECTIVES` — conformance matrix, `$K`/`$V`, unsupported directives
+2. `v-TP3-DOSFILES` — file I/O under a real DOS filesystem
+3. `v-TP3-CALCHELP` — full CALC Help and heap path
+4. `v-TP3-ARGS` — raise or document the 16-argument limit
+5. `v-TP3-CORE` — finish the statement/declaration split
+
+`SUMMARY.md` is the authoritative status document and carries the detailed
+roadmap.