# TP3-comp — state summary Recreating **Turbo Pascal 3.0** (shell / editor / compiler / interpreter) in **GNU Modula-2** (`gm2 -fiso -Wall`). Faithful to the original: screen layout, keys and behaviour are reconstructed from the disassembled TP3.0 source in `Resources/turbopascal3source/TP3/` (TPSRC1-10) and the reference manual, not guessed. ## Milestones | Milestone | Tag | State | |---|---|---| | TP3.0 main-menu shell | `v0.1-shell` | done | | WordStar-style editor | `v0.2-editor` | done | | Shell/editor polish, Ctrl-K-D / Ctrl-K-X quit | `v-TP3-SHELL-EDITOR-QUIT` | done | | Compiler skeleton + build recipe | `v-TP3-SHELL-COMPILES` | done | | Parser `Skip` bug class (9 sites) | `v-TP3-PARSER-FIXES` | done | | Standard procedures + `rel16` fix | `v-TP3-STDPROCS` | done | | Runtime library + 8086 execution harness | `v-TP3-RUNTIME-BLOB` | assembled, **never run** | | Linker + `CmdRun` | — | **not started** | ## Build ```sh cd shell && make # → shell/tpshell ``` Whole-program link **must** be two-phase; a single `gm2 -o` pass 3 silently caps identifier/error counts on a compiler-sized program: ```sh gm2 -fiso -c Compiler.mod # phase A gm2 -fiso -fgen-module-list=modules.lst -o /dev/null \ Compiler.mod Term.o TextBuf.o Posix.o Editor.o # phase B1 (rc=1 expected) gm2 -fiso -fuse-module-list=modules.lst -o tpshell \ Shell.mod Compiler.mod Term.o TextBuf.o Posix.o Editor.o # phase B2 ``` Current clean build: `make clean && make` → rc=0, `tpshell` **120792 bytes**. The one diagnostic is `./Compiler.mod: ParseExpr: too many errors in pass 3`, which is the expected phase-1 rollup that the recipe tolerates — not a real error. `shell/build_tpshell.sh` and `shell/Makefile` are authoritative. ## What is verified, and how Nothing here is "it compiles clean" — each claim below comes from a run. ### Compiler front end — `shell/tests/run_compile_tests.sh` `tests/CompileTest.mod` links `Compiler` + `TextBuf` + `Posix`, reads fixture paths from **stdin**, loads each exactly the way `LoadWorkFile` does (LF→CR normalisation, `^Z` ends the text), calls `Compile`, and prints a verdict plus a source excerpt with a caret at `errPos`. No pty, instant. ``` cd shell && tests/run_compile_tests.sh # all fixtures cd shell && tests/run_compile_tests.sh /some/dir # another fixture set printf '@dump\ntests/fixtures/t19_int1.pas\n' | ./compiletest # hex-dump the image ``` **17 of 23 fixtures compile**, up from 1 (the empty program) when the direct harness was first built. Compiling: `t01` minimal · `t04` var+assign+`writeln` · `t06` two args · `t07` 10 assignments · `t08` const · `t09` if/then/else · `t10` while · `t11` for/to · `t12` repeat/until · `t13` procedure + value param · `t15` label + goto · `t16` `writeln('a')` · `t18` bare `writeln` · `t19` `writeln(1)` · `t20` 3 string args · `t21` mixed args · `t22` `case` with two labels. Failing, all deliberately: `t02/t03/t05/t17` multi-char string literals and `t14` `array [1..5] of integer` → `ENoLib` (102), the original's "not implemented" path; `uierror` is a deliberate syntax error used by the UI test. ### Shell + editor UI — `shell/tests/uitest.py` Drives a real pty (`tests/ptyharness.py` supplies read-until-quiet, key sending and a small VT100 emulator) and asserts the TP3 **compile-error jump**: `W` load → `C` compile → `ESC` → editor opens with the cursor on the error position → `Ctrl-K D` back to the menu → `Q` exit 0. **10/10 pass**, and the reported error number is 41 (not 0 — this is what caught the `errNo := errNo` self-assignment where `Compile`'s formal shadowed the module variable). The jump mirrors original TP3 `kcwait` + `editor2` (TPSRC5:333-336, :919): `BX:=txerrpos; DEC BX; JMP editor2`, and `editor2` does `ADD BX,txbeg; INC BX` — the `DEC`/`INC` cancel, so the net is `txbeg+txerrpos` = our 0-based `errPos`. Armed as a sticky position (`Editor.GotoOffset`) rather than by changing `Run`'s signature, so `Editor.def` stays additive. `uitest.py` uses a fixture with a deliberate **syntax** error (`x := 1 + ;` → error 41, line 9 col 12) rather than a missing library feature: the latter move as the compiler grows, and a test whose expectations drift with it stops being a test. ## The executor problem (blocking everything downstream) The whole point of a Pascal→8086 compiler is that the output *runs*. Until this milestone no compiled image had ever been executed, so the plan was: get a real CPU emulator, run the image, assert the exact stdout bytes. **Unicorn 2.1.4 cannot be used for this.** `UC_MODE_16` mis-decodes 16-bit ModRM memory operands. The measurement, by loading a byte-pattern image so a load reveals its own effective address, then executing a single `LEA AX,[r+disp8]` and reading AX back with every register set to a distinct value: ``` mod=01, disp8=4 got 8086 says 8D 40 04 LEA AX,[BX+4] AX=0d04 0x504 (= BX+SI+4) WRONG 8D 41 04 LEA AX,[BX+SI+4] AX=0e04 0xd04 WRONG 8D 42 04 LEA AX,[BX+DI+4] AX=0f04 0xe04 WRONG 8D 43 04 LEA AX,[BP+4] AX=1004 0x704 WRONG 8D 45 04 LEA AX,[DI+4] AX=0904 0x904 right 8D 46 04 LEA AX,[BP+4] AX=0704 0x704 right 8D 47 04 LEA AX,[DI+4] AX=0504 0x904 WRONG mod=00 / mod=10 direct disp16 8B 1E 00 20 MOV BX,[2000] BX=1234 right 8D 06 34 12 LEA AX,[1234] AX=1234 right ``` So `rm=5` and `rm=6` decode correctly but `rm=0,1,2,3,7` do not, and only *base-register-free* addressing (direct `disp16`) is trustworthy. That rules Unicorn out as an oracle for exactly the instruction forms generated code is made of — `LEA AX,[BP+d]`, `MOV AX,[SI+d]`, `LODSW`-style loops, everything with a frame pointer. It cannot distinguish "my codegen is wrong" from "the emulator is wrong", which makes it worse than no emulator at all. `pip install --upgrade unicorn` resolves to the same 2.1.4, so this is not avoidable by upgrading. Also ruled out, for the record: - **DOSBox-X 2025.02.01** (installed, `/usr/bin/dosbox-x`, plain root-owned ELF, not a snap) starts cleanly headless under `SDL_VIDEODRIVER=dummy SDL_AUDIODRIVER=dummy`, but its `-c`/autoexec commands never observably execute: a `md` never appeared on the host, and a `.COM` that creates `OUT.TXT` via `INT 21h AH=3Dh/40h` never produced the file. Shell `>` is intercepted by dosbox-x's own wrapper (`SHELL:Redirect output to out.txt`). A `.BAT` route timed out. - **qemu-system-i386 is installed** (`/usr/bin/qemu-system-i386`) and is the next candidate: a 512-byte boot sector can load the image and a `INT 21h` shim can hand the output back. Not attempted yet. `tests/rt_exec.py` is the harness, written against Unicorn, and it is written to survive the switch: it loads the runtime, calls each entry with a known argument, and compares the bytes sent to `INT 21h` against expectations. It currently **fails**, and the failures are the emulator's, not the library's — `wrint` prints `-` for every value because `MOV AX,[BP+4]` reads the wrong address. Do not read those results as a verdict on the runtime. ## Components ### Shell — `shell/Shell.mod`, `Term.mod`, `Posix.c`, `TextBuf.mod` Exact TP3.0 screen (`Logged drive`, `Active directory`, `Work file`, `Main file`, `Edit Compile Run Save` / `Dir Quit compiler Options`, `Text: n bytes`, `Free: n bytes`, `>`), command letters drawn bold. Keys `L A W M E C R S D O Q`; any other key redraws the menu, like TP3. `W` auto-`.PAS` with `Loading`/`New File`; `S` writes `^Z` EOF and rotates the old file to `.BAK` (unlink+rename, TP3's order); `D` is a DOS `*.*` glob listing with `k bytes free`; `O` is the options submenu; `Q` confirms and prompts to save when the text changed. `E` and `C` are wired. **`R` is a stub** — it prints "Interpreter pending". ### Editor — `shell/Editor.mod` WordStar-style full-screen editing over `TextBuf`. Status line `Line n Col n Insert/Overwrite Indent X:FILENAME`; text on rows 2-24. Movement (`^S/^D/^E/^X/^A/^F/^R/^C/^W/^Z`, `^Q S/D/E/X/R/C/B/K/P`, arrows, PgUp/PgDn/Home/End), editing (`^V` insert/overtype, `^G` delete char, backspace joins lines, `^T`/`^Y` word/line, `^Q-Y` to EOL, `^N`/CR break, TAB auto-indent to the word start above), block (`^K B/T/H` mark, word, show, `^K C/V/Y` copy/move/delete, `^K R/W` file read/write), search (`^Q-F`, `^Q-A`, `^L` repeat; options B/G/n/U/W, N = no-confirm; `^A` any char, CR LF matches a line break), `^P` literal control char, `^U`/ESC aborts a prompt. `^K-D` returns to the shell with the text still in memory and `changed` reported. Disk format matches TP3: CRLF plus trailing `^Z`, load normalises, save re-expands. Detail: `TP3-EDITOR.md`, `TP3-EDITOR-PSEUDOCODE.md`. ### Compiler — `shell/Compiler.mod` Single-pass Pascal → 8086, following TPSRC6 `turbo` / TPSRC7-10: one pass over the shared `TextBuf` emitting machine code into `cbuf`, a patch list for forward references, TP3-style error reporting (number + relative position), and code/data size accounting. Emitted image is a byte array (mode word, CS/DS, size words, `CALL initmem`, `MOV BP,SP`, generated code). Working subset (v0.4): integer/char/boolean/byte scalars, constants with folding, globals, locals, value parameters, procedures and scalar-result functions, `ARRAY[const..const]` with constant indexing, control flow, `GOTO`/`EXIT`, and the standard procedures `WRITE`, `WRITELN`, `READ`, `READLN`, `HALT`. **Standard procedures** are `KBuiltin`, not `KProc`, because they are not called generically. TP3 (TPSRC8 `pwriteln`/`pwrloop`/`prdtyped`) does *not* pass a descriptor to the runtime: it inspects each argument's class and emits a *different call per type*, so formatting is fixed at compile time and the runtime only ever sees a value. `IoCall` mirrors that — one call per argument, then a final call for the line break: ``` writeln(1) MOV AX,1 ; PUSH AX ; CALL 20H ; ADD SP,2 ; CALL 40H writeln('a') MOV AX,'a' ; PUSH AX ; CALL 28H ; ADD SP,2 ; CALL 40H readln(x) LEA AX,[0104]; PUSH AX ; CALL 48H ; ADD SP,2 ; CALL 60H ``` `READ`/`READLN` push the *address* so the runtime can store (`EmPushVarAddr`: `8D 46 disp` / `8D 06 off`); a non-variable argument is `ETypeErr` (56), as in TP3. `TU_WrInt/Char/Bool/Real`, `TU_WrLn`, `TU_RdInt/Char/Bool`, `TU_RdLn`, `TU_Halt` continue the existing `TU_*` image-base space (`TU_InitMem=8H`, `TU_ProgEnd=10H`, `TU_StackChk=18H`). Detail: `TP3-COMPILER.md`. ### Runtime library — `shell/Runtime.mod`, `shell/Runtime.def` The 8086 runtime, **assembled byte by byte from Modula-2** — no external assembler and no checked-in binary, so the tree stays self-contained and the entry offsets are *derived* rather than guessed. Each emitter is one instruction with its ModRM byte spelled out in a comment so the encoding can be checked by hand against an 8086 table. This is the same approach `Compiler.mod` already takes (`Ebyte`/`Eword`/`EmCall`). It mirrors the original's own mechanism: TPSRC7 `copyrt` copies the runtime into the front of the code buffer (`SI=DI=0`, `REPZ MOVSB`) and `pc` is then initialised past it (`MOV pc,#$2D7C`). Same shape here — the compiler is meant to copy the blob to the front of `cbuf` and start `pc`/`dc` past it. Runtime data therefore lives at fixed low offsets and needs no relocation, and because both sides of every `CALL` shift by the same amount, `EmCall`'s displacement arithmetic is unaffected by the runtime being prepended. Current blob: **366 bytes**, 13 entries, offsets read back out of the assembled bytes by `tests/RtProbe.mod`: | entry | offset | entry | offset | entry | offset | |---|---|---|---|---|---| | `initmem` | 0 | `wrint` | 36 | `rdint` | 142 | | `progend` | 28 | `wrchar` | 97 | `rdchar` | 243 | | `stackchk` | 35 | `wrbool` | 106 | `rdbool` | 264 | | `halt` | 28 | `wrreal` | 126 | `rdln` | 310 | | | | `wrln` | 134 | | | `progend` and `halt` deliberately share one address (`XOR AX,AX / MOV AH,4C / INT 21h / RET`): the compiler already zeroes AX before `progend` and discards the `HALT` argument at compile time, so both leave with exit code 0. Conventions, matching `Compiler.IoCall` exactly: | entry | argument | notes | |---|---|---| | `WrInt/WrChar/WrBool/WrReal` | one 16-bit **value** on the stack | caller pops | | `RdInt/RdChar/RdBool` | one **address** on the stack | caller pops | | `WrLn/RdLn/StackChk` | nothing | | | `InitMem` | `AX` = offset of the program header | a *register*, not a stack word | | `ProgEnd/Halt` | nothing | exits, code 0 | `InitMem` reads the data base and end out of the header words at `+2`/`+6` and zeroes that range, because Pascal leaves globals undefined. `StackChk` is a bare `RET` — range and stack checking aren't compiled in yet, and the call site sits mid-expression, so it must not touch a register. Two label-name plus fixup list: `rel8`, `rel16` and runtime-data addresses are all patched after the blob is placed, so nothing depends on a hand-computed displacement. **It has never executed.** See the emulator finding below. ## Honest limitations - **A compiled image still cannot be executed.** `CmdRun` is a stub, the linker is unwritten, and no 8086 executor on this machine has yet proved trustworthy (above). The emitted code is verified *byte by byte* against the offsets the compiler intends, but nothing has ever run it. - **The runtime is written but unproven.** `Runtime.mod` assembles to 366 bytes and its entry offsets are derived from the emitted bytes, but it has never been executed on a correct CPU, so treat every encoding in it as unverified even though three of its own bugs were caught by running it under a broken emulator. - **The compiler is not yet wired to the runtime.** `Compiler.mod` still carries the hardcoded placeholder `TU_*` constants (`TU_InitMem=8H`, `TU_ProgEnd=10H`, …) and still starts `pc` at 0, so emitted images do not contain the runtime and those offsets are still wrong. `Runtime.mod` is not in `make` or `run_compile_tests.sh` yet for the same reason. Note the prologue's `CALL TU_InitMem` targets offset 8, which is currently the `hdrMax` header word — coherent only once the blob is really prepended. *(Correction to the earlier note in this file: the `TU_InitMem=8` "collision" was a false alarm. Per TPSRC7 the runtime is copied to the *front* of the code buffer and `pc` starts past it, so `TU_*` offsets are runtime-relative, not image-absolute, and no rebasing of the displacement arithmetic is needed.)* - **No string runtime.** `RdConst` gives a 1-character literal as `TScalar` (its char code) and only longer literals as `TString`, so multi-char literals raise `ENoLib`. `writeln('a')` works via a `chr` flag on `ERes`; without it the compiler emitted the *integer* writer and would have printed 97 while the test still said OK. - **Not implemented** (all `ENoLib`): real, set, record, file, string, and any type wider than 2 bytes. `with` is `ENoLib`. `case` *is* implemented (cascade `CMP`/`JNZ` per label, per RESUME-TP3.md §3.6) but only over scalar labels — subrange labels and label lists are untested. - gm2 string-literal → `ARRAY OF CHAR` assignment copies the literal *plus a NUL* and leaves the tail untouched, so NUL-terminated tables are safe — this was checked, not assumed. ## Three bugs only a byte-level dump could find None of these produced a compile error, and two produced *passing* tests. All three were found by hex-dumping the emitted image and decoding it by hand — code sizes looked perfectly plausible throughout. 1. **`rel16` off by 2 in every direct CALL/JMP.** `EmCall`/`EmJmpNear`/ `EmJcc` computed the displacement from `pc` at a point where `pc` already pointed past the opcode and *at* the displacement field; x86 measures from the end of the instruction (`pc + 2`). `ResolvePatches`, on the forward-patched path, was already right — which is why forward gotos looked fine and backward ones did not. 2. **`EmMovAxSp` emitted `8B 04` with no SIB byte.** ModRM `04` means "a SIB byte follows", so the `CMP AX,imm16` of the *next* instruction was eaten as that SIB byte, and the intended `MOV AX,[SP]` became `MOV AX,[BP+DI+disp]`. Every `case` label comparison therefore vanished and `case` compiled to a chain of loads from garbage addresses — while the fixture reported OK. Correct encoding is `8B 44 24 00` (`[SP]` cannot use mod=00, that computes `BP+SP`). 3. **The dump tool's own offset column was wrong.** It printed 4 *nibbles* through a helper that formats a **byte**, so every row was labelled 16× too large (`0010` shown as `00000100`). A misleading tool is worse than none — it corrupts any offset arithmetic done from its output. ## Bugs found by actually running code Executing the hand-assembled runtime — even under a broken emulator — paid for itself immediately, because a wrong encoding *executes* rather than failing to assemble. Four, none of which a compiler diagnostic would ever have reported. 1. **`B()` silently truncated multi-byte opcodes.** `PROCEDURE B` emits exactly one byte and masks with `MOD 100H`, so `B (8BE4H)` — a two-byte opcode passed as one literal — emitted just `E4`. `MOV BP,SP` was missing from every frame in the runtime, so `BP` stayed 0 and *every* `BP`-relative access read address 4. Found by decoding the hex dump; the byte is gone, not wrong. 2. **`MOV BP,SP` encoded as `8B E4`, which is `MOV SP,SP`** — a no-op. The ModRM byte is `mod·64 + reg·8 + rm`, and I mis-derived it. `Compiler.mod`'s `EmMovBpSp` had it right all along (`8B 0CH`); only the new module was wrong. This is why the encoding is now written as three explicit `B` calls with the arithmetic in a comment rather than as one hex literal. 3. **`InitMem` read its argument from `[SP]`** — the return address. The convention is a register (`AX`), unlike the per-argument I/O entries which do take a stack word. Caught because the data area was never cleared. 4. **`EmPushVarAddr` computes the wrong base register for locals** (pre-existing, **not yet fixed**). It emits `8D 46 disp`, which is `LEA AX,[SI+disp8]`, but intends `LEA AX,[BP+disp8]` = `8D 45 disp`. So `read` into a *local* variable has always addressed the wrong cell, silently. It also truncates the offset with `off MOD 100H`, losing displacements above 255. The global form `8D 06 off` (`LEA AX,[disp16]`) is correct. Found while writing the runtime's read entries, which needed the same encoding to be right. ## gm2 / ISO Modula-2 pitfalls hit along the way - **Two-phase link** (above) — a single whole-program pass 3 caps identifiers/errors silently. - `EXIT` inside the program-header parameter `WHILE` **ICEs gm2** in pass 3 (`ExitStatement → PopExit → M2StackWord_PopWord → invalidloc`). Extracting the loop into its own procedure did *not* help. Use a `BOOLEAN` "advanced" flag, never `EXIT`. (`LOOP`+`EXIT` is fine, and is used widely.) - A bare `HALT` aborts under `-fiso` (SIGABRT, exit 134); `HALT (0)` is correct. - `CHAR` is not the ZType: `ch = 09H` must be `ORD (ch) = 09H`. - gm2's ISO `SYSTEM` exports `ORD` but **not** `Ord` — the import is case-sensitive here despite gm2's usual case-insensitivity, so `FROM SYSTEM IMPORT Ord` fails with "unknown symbol" while plain `ORD (c)` compiles. Write `ORD`, never `Ord`. - ISO forbids dropping a function result in a statement → the `DropCh`/ `DropB`/`DropC` discard helpers wrap ~39 call sites. - `AND`/`OR`/`NOT` on 16-bit `CARDINAL` → `BitAnd`/`BitOr`/`BitNot`. - `PROCEDURE f : T` is invalid; the file's convention is `PROCEDURE f () : T`. - ISO will not index a plain string constant as an array (needs an array constructor) — compute hex digits with `CHR` instead. - Foreign modules (`FOR "C"`) must use `ADDRESS`, not Modula-2 pointer types; cast at the call site. `Posix` has no `Posix.mod` (do not try to rebuild it) and no `argv` binding. - termios raw mode: every newline is an explicit `CR LF`. - Foreign modules and `Posix` cannot pass `argv`; the test harness reads fixture paths from **stdin** instead. ## Reference material - `TP3-COMPILER.md` — the compiler: what was changed, the `Skip` bug class, the `rel16` off-by-2, standard procedures, current matrix. - `TP3-EDITOR.md` / `TP3-EDITOR-PSEUDOCODE.md` — the original TP3 editor, from TPSRC5/TPSRC6. - `RESUME-TP3.md` — book-derived reference for the 8086 code TP3 generates per Pascal construct (types, skeletons, arithmetic, IF/CASE/REPEAT/WHILE/ FOR, procedures, parameters, functions, I/O, typed constants, absolutes), plus the `TU_*` runtime entry list. - `Resources/turbopascal3source/TP3/` — TPSRC1-10, the disassembled original. **This is the ground truth**; when our behaviour and the book disagree, the disassembly wins. - `Resources/coeur-tp-ocr/` — OCR of "Au coeur de Turbo Pascal". ## Next steps 1. **Get a trustworthy 8086 executor**, because it gates items 2–4 and nothing else can be claimed until a compiled image runs. `qemu-system-i386` with a boot-sector loader and an `INT 21h` shim is the plan; keep `rt_exec.py`'s expectations and change only the machine behind them. (Do *not* reach for Unicorn's 16-bit mode again, and do not re-derive the `rm` table by hand again — check it against a real decoder.) 2. **Fix `EmPushVarAddr`** — `8D 45`/`8D 85` for locals, full `disp16`, per bug 4 above. Two lines, and `read` into a local is wrong until it is done. 3. **Wire the runtime in and write the linker**: `pc := RT_Size`, `dc := RT_Size + 1000H` (a fixed 4 KiB code/data gap, so a program's data can't collide with its code in a single 64 K `.COM` segment), take the `TU_*` offsets from `RT_Entry` instead of the hardcoded constants, patch the header words (`hdrDS` = data base, `hdrHeap` = data end, so `InitMem` can zero globals), pad the image to cover the data area, and emit the `.COM`. Add `Runtime` to the `make` and `run_compile_tests.sh` rebuild lists. 4. **Prove it end to end**: compile a fixture, link, execute, assert the exact stdout bytes (`writeln('hi')` → `hi`). That single assertion is what turns this from "assembles" into "works". 5. **`CmdRun`** — run the emitted image from the `R` menu key. 6. **String runtime** — unlocks the last 4 real fixture failures. 7. Nested procedures / recursion, `var` parameters (the `SEG:OFF` push from RESUME-TP3.md §3.11), range/index checks (`TU_RANGE_CHECK`, `TU_INDEX_CHECK`), typed constants (RESUME-TP3.md §3.14). 8. Harden the program-header parameter loop against non-advancing input (`program p(1;)`) with a `BOOLEAN` flag — **not** `EXIT`, which ICEs gm2.