# TP3-comp — Turbo Pascal 3 clone in GNU Modula-2 (ISO) **Status:** compiler-core (`Compiler.mod`) + shell compiled clean with `gm2 -fiso -Wall`, and the ISO shell `tpshell` builds and runs (ESC-key driven, TP3-style DOS screen redraws). This is the working *implementation* companion — the human overview is `TP3-COMPILER.md` history summary and `RESUME-TP3.md`. ## Build (proven, two-phase — the ONLY recipe that beats GNU gm2's pass-3 "too many errors / too many identifiers" cap on compiler-sized whole-program programs) ``` gm2 -fiso -c Compiler.mod # phase A: small per-module pass gm2 -fiso -fgen-module-list=modules.lst -o /dev/null \ Compiler.mod Term.o TextBuf.o Posix.o Editor.o # phase B1: build list gm2 -fiso -fuse-module-list=modules.lst -o tpshell \ Shell.mod Compiler.mod Term.o TextBuf.o Posix.o Editor.o # phase B2: link ``` (Equivalent make target `tpshell` in `shell/Makefile`.) The single `gm2 -o` whole-program pass 3 silently caps identifier/error counts — see `build_tpshell.sh`.) ## What was changed inside `Compiler.mod` (all ISO-legal, gm2-verifiable) 1. **CHAR vs hex-int comparisons** (lines ~829, ~1086): Modula-2 `CHAR` is not the ZType; `ch = 09H` → `ORD (ch) = 09H` (and likewise 0DH/0AH/0CH). 2. **Return-value discard** (ISO forbids drop in statements; gm2 treats as error and has no `-Wno-`): added `DropCh/DropB/DropC` discard helpers and wrapped ~39 statement call sites of `GetCh`, `MatchKey`, `NewSym`, `EmCall`, `EmJmpNear` (no output reground clutter; TP3 TP5 `cascadepos` `waitkey` semantics preserved via the editor's ESC wait). 3. **`AND`/`OR`/`NOT` on W16-folded CARDINAL** (lines ~1233/1246/1454) are illegal on ZType in gm2 → added `BitAnd/BitOr/BitNot` (BITSET `*`/`+`/`-` fold, VAL-projected back to LONGINT) and rewrote the operands. 4. **`Compile` error-return params** aligned to `Compiler.def` (`errNo`/`errPos`, was `eNo`/`ePos`) — gm2 ISO rejects a disparate name in the proper procedure. 5. **`HALT` -> `HALT (0)`** in `Shell.mod`. A bare `HALT` (no operand) aborts under `gm2 -fiso`: a minimal repro exits 134 (SIGABRT) with a bare `HALT` but 0 with `HALT (0)`. Quitting the shell died on SIGABRT. 6. **Module-level `errNo` renamed `errNum`.** `Compile`'s formal `errNo` shadowed the module variable, so `errNo := errNo` was a *self*-assignment and every error code reached the caller as 0. The compiler's whole-program 8086-emit, TEXT-equivalent, pastres and errexit flow are otherwise unchanged (mirrors TP3 TPSRC file, ConvertTP3). ## Lexer/parser bug class: "Alpha test without Skip" The single largest source of wrong behaviour. `Skip ()` is what folds blanks, `{ }` and `(* *)` comments, and `CR`; neither `PeekKw` nor `MatchKey` leaves the cursor past it (`PeekKw` save/restores `srcPos`, `MatchKey` stops right after the word it matched). So a routine entered *immediately after a keyword* sees the CR/blank that follows that keyword, and any ``` IF NOT Alpha (CurCh ()) THEN Err (EUnknown) ; RETURN ``` fires with `EUnknown` (41) pointing at a blank. That is why every program used to fail on `program`'s program name, and why `writeln(...)` failed on its `(`. Fixed by adding the missing `Skip ()` at each entry point (it is idempotent, so it is safe even where a `Skip` already ran): | routine | entered after | |---|---| | program header | `PROGRAM` | | `DefVar` | `VAR` | | `DefConst` | `CONST` | | `DefType` | `TYPE` | | `ParseType` | `:` / `=` | | `ProcFunc` (name) | `PROCEDURE` / `FUNCTION` | | `ProcFunc` (parameter name) | `(`, or a var-parameter's `VAR` | | `Statmnt`, `TkFor` branch | `FOR` | | `Statmnt`, `TkGoto` branch | `GOTO` | `DefLabelPart`, `MatchKey`, `MatchDelim` and `MatchAssign` already had theirs. Three related defects fixed alongside: - `ProcFunc`'s parameter loop used `MatchKey (tok) AND (tok = TkVar)` to detect a var-parameter. `MatchKey` **consumes** the word it reads, so using it as a lookahead ate the parameter's own name. Replaced with `PeekKw` (non-consuming), consuming with `DropB (MatchKey (tok))` only on a `VAR` hit. - `ProcFunc` called `ParseType` straight after the parameter name without consuming the `:` of `name : type`; added `ExpectDelim (':', ENoSemi)`. - `ParseLabelStmt` consumed `n :` but never parsed the statement the label is attached to, so `Compound` then demanded a `;` that does not exist in `1: x := 1`. `Statmnt` now parses the labelled statement. ## Tests Several harnesses live in `shell/tests/` -- in the repo on purpose, because `/tmp` is wiped between sessions and an earlier /tmp-only harness was lost with it. `shell/tests/run_all.sh` runs them all; its header says which is which. `shell/SUMMARY.md` (at the repo root) is the state-of-the-project document. ### `run_compile_tests.sh` -- compiler front end, no pty, instant `tests/CompileTest.mod` links `Compiler` + `TextBuf` + `Posix` and reads fixture paths from **stdin**, one per line. For each it does exactly what the shell's `LoadWorkFile` does (LF -> CR normalisation, `^Z` ends the text), calls `Compile`, and prints a one-line verdict plus a source excerpt with a caret under the error position. It rebuilds `Compiler.o` when the source is newer, so an edit is picked up automatically. ``` cd shell && tests/run_compile_tests.sh # all fixtures cd shell && tests/run_compile_tests.sh /some/dir # another fixture set ``` **The fixture matrix is not written here.** It used to be, and it went stale the moment it was copied -- a second copy of a table is a table that lies. The live one is `shell/tests/fixtures/expected.tsv`, which the runner *asserts* against: per fixture it pins the verdict plus the code size plus the data size (or the error number plus the error position), and the rationale for every re-baselining is written in the file rather than in a commit message. Read that for what compiles; this document for how. The two fixtures that fail are `t14` (`array [1..5] of integer` at its point of use) and `t25` (a string literal used as a value, `s := 'hi'`), both `ENoLib` (102) -- the original's own "not implemented" path, and pinned on purpose so that implementing them is a deliberate change to a test rather than an accident. `uierror` is a deliberate syntax error used by `uitest.py`. ### `run_com_exec.py` -- the emitted image, on a CPU The byte-level checks above all stop at "the compiler produced what it intended to produce", which is where this project's bugs live: a wrong ModRM byte is a perfectly well-formed instruction that does something else, so no amount of inspecting the compiler's intent will catch it. This harness closes that gap the only way it can be closed -- by running the thing. For each of 21 fixtures it compiles a `.COM`, writes a 1.44 MB floppy with the image in it, boots `qemu-system-i386 -fda disk.img -serial out.txt`, and compares `out.txt` to the committed `tests/fixtures/tNN.out` **exactly** -- CRLF included -- plus the exit code the program hands to `INT 21h AH=4Ch`. ``` cd shell && python3 tests/run_com_exec.py # all 21 cd shell && python3 tests/run_com_exec.py --show # print the serial file cd shell && python3 tests/run_com_exec.py --rebless # rewrite the .out files ``` `--rebless` exists because 21 golden files will eventually need re-basing, and re-basing by hand is how a red test is quietly made green. The rule this project follows instead: **re-baseline deliberately, never to turn a red test green.** If a fixture's expected output changes, the reason goes in `expected.tsv` or in `SUMMARY.md` first. `shell/tests/exec/bootcom.s` is the boot sector: 512 bytes, reads the `.COM` off the same floppy with `INT 13h`, sets `DS=ES=0`, `SS=2000h SP=2004h`, builds the three GDT descriptors a `.COM` expects, installs an `INT 21h` shim for `AH=02h/09h/08h/4Ch` to the serial port, and `JMP 0000:0100`. It is reusable rather than duplicated, and `rt_exec.py` reuses it. It reads the sixteen sectors to `8000h` and then `REP MOVSW`s them down to `0100h`, which is the whole boot sector's one subtlety. A direct DMA to `0100h` is what DOS does, and it *works* -- sets CF=0, returns the right sector count -- but `0400h-04FFh` is the BIOS data area, where SeaBIOS keeps live state that it writes back *after* the transfer, and ten bytes of BIOS data end up on top of the image. The image is the right length in the right place and ten bytes of it are wrong, which is the worst shape a failure can have: a partial read or a wrong sector count would have been visible, this was not. `8000h` is clear of the IVT, the BDA, SeaBIOS's stack at `0700h` and the ROM window at `C000h`; and the copy is executed code, so it is the last thing that touches the image and no BIOS call follows it. **21 of 33 fixtures execute.** 3 do not compile, and 9 compile without a `.out` file -- and those 9 are the control-flow fixtures (`t08` const, `t09` if, `t10` while, `t11` for, `t12` repeat, `t13` procedure, `t15` label, `t27` five locals, `t28` 70 parameters). They have byte-level checks and no behavioural check. That is the next gap to close, and it is a gap in the *tests*, not in the compiler. What execution found that no byte-level check could: the `FOR` loop was post-tested, `EXIT` targeted the increment, procedure bodies executed as part of the main body, and the procedure-skip jump's patch slot could not be distinguished from "no jump" because slot 0 is legitimate. All four had plausible byte counts. See `SUMMARY.md` for the full list. ### `rt_exec.py` -- the runtime entries, called one at a time `run_com_exec.py` asks whether *programs* behave. This asks whether the **runtime entries** do, which is a different question: a compiled program never calls `rdint` without a `readln` behind it, never calls `rdbool` twice in a row, and never exercises the `INT16` extremes one at a time. It is in `run_all.sh` and **36/36 pass** — 35 cases plus the pre-flight below, which is counted as a check because it is one. ``` cd shell && python3 tests/rt_exec.py # all 35 cases + the pre-flight cd shell && python3 tests/rt_exec.py --probe # the RtProbe dump alone cd shell && python3 tests/rt_exec.py --list # the case list, with expectations cd shell && python3 tests/rt_exec.py --show # print a case's wire bytes cd shell && python3 tests/rt_exec.py 3 --show # one case ``` **One boot per case, deliberately.** A machine that has already run a case has already run a runtime entry, and if that entry corrupted something the next case inherits it. About a tenth of a second a boot buys a machine that has provably never executed anything, which is the only version of that claim worth making. **The case record is stated twice on purpose.** `rt_exec.py` and `exec/rtdrv.s` each describe the same 40-byte record independently, and a case only passes if the two agree on every offset. A driver that read the wrong field would still boot, still run, and still print, so the duplication *is* the check -- the same argument as the `.COM` layout constants in `run_com_tests.sh`. **`--probe` is the single recipe for `RtProbe`.** `run_all.sh` used to hand-build a second copy at `/tmp/tp_rtprobe`; two recipes for one artifact is how a stale binary gets believed, and a probe older than the runtime it reports on is worse than no probe, because its failure -- a blob with the wrong addresses baked into it -- is invisible at the byte level and catastrophic at the behaviour level. `ensure_rtprobe()` mtime-checks `Runtime.mod`, `Runtime.def`, `Runtime.o`, `Posix.c` and `RtProbe.mod`. What it found, that nothing else had: - **`CmpAl (20)` was decimal twenty.** `rdint`'s lead-in skip is `CmpAl (20H)`; written `20` it emitted `3C 14`, so a leading space was never skipped and the scan ended having read nothing. Every magic number in `Runtime.mod` is now hex, because a literal that *reads* like hex and is decimal is silent. - **`wrchar` and `wrbool` destroyed BP.** Both borrow it to reach their argument -- `[SP]` cannot be encoded in 16-bit mode -- and neither saved it. BP is the one register an entry may keep, because the driver keeps its cursor into the case record there. `wrchar` therefore sent the *next* call to a garbage address, the machine triple-faulted, and the run printed the record header twice and hung. The second one is the interesting one, because **every byte-level check had called that shape correct.** The bytes were well formed, the size did not change, the golden matched, all branch targets were on instruction boundaries, and the name audit said every helper emitted what its name said. Worse, `check_runtime.py`'s entry golden had *blessed* it in five bytes. A positional golden blesses whatever is there. The rule is now stated in two places that can disagree: the entry goldens carry the `PUSH BP` and the `POP BP` (what the bytes must be), and `rt_exec.py`'s `check_bp_contract` pre-flight checks the same rule over the built blob before any machine starts (what the bytes must *mean*). Its two halves are not equally strong and the code says which is which -- "must start with `55 8B EC`" is exact, and "must contain a `5D`" is a *screen*, because `5D` is also a displacement byte and this code does not disassemble. Demanding the `POP` be contiguous with something else is not an improvement: `wrbool` legitimately closes its frame after the `INT 21h`, and a rule insisting on `89 EC 5D` fails a correct entry, which is worse here than a miss, because it teaches a reader to distrust the check. And it changed the emitters, not just the tests. Reaching the argument at `[BP+2]` versus `[BP+4]` is a one-byte difference that reads as a plausible character, and the first version of the fix passed that displacement as a Modula-2 **parameter** -- which is invisible to `audit_helpers.py`, because the audit reads the *name*. The emitters are `MovAlArg2`/`MovAlArg4` and `CmpArg2W0`/`CmpArg4W0` now, displacement in the name, so the audit pins it: a `CmpArg4W0` that emitted the `+2` form fails with *"step 2: displacement is 2, name says 4"*. That is the general lesson -- **a Modula-2 parameter is the one thing a name-based audit cannot see, so a thing the audit must check has to be in the name.** ### Hex-dumping the emitted image A line reading `@dump` on stdin switches on a hex dump of the emitted 8086 code (via the new `Compiler.CodeByteAt`), which is how the `rel16` off-by-2 below was found -- sizes alone could never have shown it: ``` cd shell printf '@dump\ntests/fixtures/t19_int1.pas\n' | ./compiletest ``` ``` 0000: 01 00 02 00 10 00 00 00 00 00 10 00 00 00 00 00 0010: E8 F5 FF 8B EC B8 01 00 50 E8 04 00 83 C4 02 E8 0020: 1E 00 33 C0 E8 E9 FF ``` which reads: header words (CS=1, DS=0x10 = 256/16, 16 open files), `CALL 8H` (initmem), `MOV BP,SP`, `MOV AX,1`, `PUSH AX`, `CALL 20H` (WrInt), `ADD SP,2`, `CALL 40H` (WrLn), `XOR AX,AX`, `CALL 10H` (ProgEnd). ### `uitest.py` -- shell/editor behaviour that only exists interactively Drives a real pty (`ptyharness.py` supplies read-until-quiet, key sending and a small VT100 emulator) and asserts the TP3 **compile-error jump**: `W` load -> `C` compile -> `ESC` -> the editor opens with the cursor exactly on the error position -> `Ctrl-K D` back to the menu -> `Q` exit 0. 10/10 pass on `uierror.pas`, and the reported error is 41 -- i.e. the `errNo` self-assignment fix is now proven through the real UI. The fixture carries a deliberate *syntax* error (`x := 1 + ;` -> error 41 at line 9, column 12) rather than a missing library feature, on purpose: the latter move as the compiler grows, and a UI test whose expectations drift with it stops being a test. The jump mirrors original TP3 `kcwait` + `editor2` (`Resources/turbopascal3source/TP3/TPSRC5:333-336` and `:919`): `waitesc; BX:=txerrpos; DEC BX; JMP editor2`, where `editor2` then does `ADD BX,txbeg; INC BX` -- the `DEC`/`INC` cancel, so the net position is `txbeg+txerrpos` = our 0-based `errPos`. It is armed as a sticky position (`Editor.GotoOffset`) instead of by changing `Run`'s signature, so `Editor.def` stays additive. Note `LoadWorkFile` ends with a `Pause`, so a driver must send one filler key after the path; skipping it desynchronises every later keypress. ## Standard procedures (the builtin table) `Inittur` defines six predefined *types* (INTEGER/BYTE/CHAR/BOOLEAN/REAL/ STRING), TRUE/FALSE and two temporaries, plus now five **procedures** via `DefBuiltins`: WRITE, WRITELN, READ, READLN, HALT. Before this, `WRITELN` was simply absent from the symbol table, `Statmnt`'s identifier branch failed its `Search`, and every program that printed anything died with `EUnknown` (41) on the `(` after the call name. They are tagged `KBuiltin`, not `KProc`, because they are not called generically. TP3 (TPSRC8 `pwriteln` / `pwrloop` / `prdtyped`) does **not** hand the runtime a descriptor: it inspects each argument's class and emits a *different call per type*, so the formatting is fixed at compile time and the runtime only ever sees a value. `IoCall` mirrors that: ``` writeln(1) -> MOV AX,1 ; PUSH AX ; CALL 20H ; ADD SP,2 ; CALL 40H writeln('a') -> MOV AX,'a' ; PUSH AX ; CALL 28H ; ADD SP,2 ; CALL 40H writeln(1,'a') -> ... CALL 20H ... CALL 28H ... CALL 40H readln(x) -> LEA AX,[0104] ; PUSH AX ; CALL 48H ; ADD SP,2 ; CALL 60H ``` `TU_WrInt/Char/Bool/Real`, `TU_WrLn`, `TU_RdInt/Char/Bool`, `TU_RdLn` and `TU_Halt` are new image-base entry constants continuing the existing `TU_*` space (`TU_InitMem=8H`, `TU_ProgEnd=10H`, `TU_StackChk=18H`). READ/READLN push the *address* of the variable (new `EmPushVarAddr`: `8D 46 disp` / `8D 06 off`) so the runtime can store; a non-variable argument is `ETypeErr` (56), as in TP3. *Superseded.* Both the `TU_*` offsets and the `8D 46 disp` encoding changed after this was written. The runtime is now prepended to the image and the `TU_*` values are assigned from `Runtime.RT_Entry` rather than written down; and `EmPushVarAddr` for locals was emitting the wrong base register (`8D 46` is `[SI+disp8]`, not `[BP+disp8]`) while truncating the displacement to a byte. `EmBpDisp` now owns that choice -- disp8 iff `off <= 127`, else disp16 -- and is shared by `EmLoadVar`, `EmStoreVar` and `EmPushVarAddr`. See `SUMMARY.md`, "Bugs found by actually running code" item 4, and `shell/tests/check_framedisp.py`. A **single-character** literal needed care. `RdConst` reports `'a'` as `TScalar` (its char code 97) and only longer literals as `TString`. That is right for `c := 'a'` but would make `writeln('a')` print **97**, so `ERes` grew a `chr` flag, set in `ParseAtom` and honoured in `IoCall`, which then picks the char entry. Without it `writeln('a')` compiled cleanly and printed the wrong thing -- a green test lying, which is worse than a failure. Still pending, and deliberately reported rather than faked: the **runtime blob** itself (`CmdRun`, the interpreter, and the linker that rebases these entry offsets by the runtime's size) and the **string** runtime, so multi-character literals still raise `ENoLib` (102). `ENoLib` is also the path for real/set/record/file and for any type wider than 2 bytes. ## Codegen bug: every direct CALL/JMP was 2 bytes long Found by dumping the emitted image, not by size. `EmCall`, `EmJmpNear` and `EmJcc` computed their `rel16` as ``` rel := (target + 10000H - pc) MOD 10000H ``` but at that point `pc` already points *past the opcode and at the displacement field* -- the instruction does not end until `pc + 2`, and x86 measures `rel16` from the end of the instruction. So every resolved-immediately branch and jump landed 2 bytes past its target. `ResolvePatches`, used for the *forward* patched path, already had it right (`target - (place + 2)`), which is why forward gotos looked fine and backward ones did not. All three now use `pc + 2`. This was silently harmless while nothing executed the image, which is exactly why it survived so long. Verified on `t10_while`: the `JZ` forward patch lands on the instruction after the loop, and the `JMP` back-edge now lands exactly on the loop head instead of 2 bytes into it -- and `t10_while` is one of the nine fixtures that *still* do not execute, so the claim is byte-level only. ## Codegen bug class: a patch slot of 0 is not "no patch" `EmJmpNear (target)` does not emit a branch -- it emits one and returns the **index of a patch slot** in the image's fixup list, because the target is not known yet when the branch is emitted. `Compile` stores that index in `overProc` and patches it later with `SetPatTgt (overProc, pc)`. The first version used `overProc := 0` as the "there was no such jump" sentinel, and **patch slot 0 is a perfectly valid slot**. So ``` IF overProc # 0 THEN SetPatTgt (overProc, pc) END ; ``` skipped the patch for any program whose procedure-skip jump happened to be the *first* patch in the image. The jump kept its placeholder target of 0, landed at image offset 0 -- which is the entry `JMP` -- and looped forever. `t31_procparam` did not fail, it **hung**. The fix is a separate `hasProc : BOOLEAN`, and the rule generalises: **a plausible placeholder is indistinguishable from a real value.** `0` as "no target", `-1` as "unbounded", `0` as "flag not set" are all correct until the first real value is 0. Reach for a `BOOLEAN`; it has no collision to have. ## Codegen: procedure bodies are emitted inside the caller's main body The compiler emits a procedure's body *between* the caller program's prologue and its own main body, so a jump over it is mandatory -- without it the main body runs straight into the procedure's code. `DeclaresProc ()` answers "does this program declare any procedure at all?" with a save/restore-`srcPos` lookahead that steps over `:` and `;` and stops at `srcLen`; `Compile` then emits `EmJmpNear (0)` after the prolog when the answer is TRUE, and patches it once `DefPart` has emitted the bodies. This is why four fixture code sizes grew by exactly 3 bytes (the `E9` plus its rel16): `t13_proc` 53->56, `t27_localvar` 122->125, `t28_farparam` 123->126, `t31_procparam` 66->69. A size change in a pinned matrix is a claim that has to be justified, and each of these is written into `expected.tsv`. `DeclaresProc` reads lookahead characters with ``` ch : CHAR ; ch := GetCh () ``` rather than a bare `GetCh ()`, because `gm2 -fiso` rejects an ignored function result -- a rule that has bitten this project in several unrelated places. ## gm2 pitfall: `EXIT` inside the program-header `WHILE` crashes pass 3 Adding an `EXIT` to the program-header parameter loop (to guard against non-advancing input on malformed input such as `program p(1;)`) ICEs gm2: ``` internal compiler error: Abandon ... ExitStatement -> PopExit -> M2StackWord_PopWord -> invalidloc ``` Extracting the loop into its own procedure did **not** help. So the header fix is deliberately exactly one added `Skip ()` and no `EXIT`. If that loop must be hardened, use a `BOOLEAN` "advanced" flag, never `EXIT`. ## gm2 / ISO trap: `AND` is not short-circuit, and a lookahead must not match Two rules that produced the same class of bug: - `IF MatchKey (tok) AND (tok = TkVar) THEN` -- `MatchKey` **consumes** the word it recognises, and `AND` evaluates both sides. So this is not a lookahead, it is a parse that eats the very token being tested for. The shape that works is `PeekKw (tok)` (which save/restores `srcPos`) for the question, and `DropB (MatchKey (tok))` only on the branch that commits. This is the "Alpha test without `Skip`" family above, one level up: there, the cursor was not past whitespace; here, it was past the token. - `IF ` may not be the last thing in a compound statement. It is a compile error under `-fiso`, and the error points at the `END`, not at the `IF`. ## gm2 trap: an implementation module must not re-declare its own `.def` CONST `Runtime.def` declares `LoadBias = 100H` because `Compiler.mod` needs it too. `Runtime.mod` must then **not** declare it again -- duplicating a CONST that the definition module already exported is an error, and the message points at the duplicate, not at the `.def`, so it reads like a redeclaration problem rather than "this already exists upstream". ## `gm2 -fiso` rejects a dropped function result `GetCh ()` as a statement, or any call whose result is discarded, is an error. Hence the `DropCh`/`DropB`/`DropC` helpers throughout, and hence `ch : CHAR ; ch := GetCh ()` in `DeclaresProc`. There are ~39 such call sites and every one of them is a place where a reader might "simplify" the `DropB` away. *Generated: 2026-09-30*