|
|
@@ -1,60 +1,263 @@
|
|
|
# TP3-comp — state summary
|
|
|
|
|
|
-Recreating Turbo Pascal 3.0 (compiler / editor / shell) in GNU Modula-2
|
|
|
-(`-fiso`), starting with the shell.
|
|
|
-
|
|
|
-## Milestone: shell (TP3.0 main-menu shell)
|
|
|
-
|
|
|
-Build: `cd shell && make` (gm2 16.0.1, `-fiso`; output `shell/tpshell`)
|
|
|
-
|
|
|
-### Files
|
|
|
-- `shell/Term.def` / `shell/Term.mod` — raw termios keyboard + ANSI screen
|
|
|
- (ClrScr, GotoXY, Marked/Normal, blocking GetCh, Beep).
|
|
|
-- `shell/Posix.def` / `shell/Posix.c` — `DEFINITION MODULE FOR "C"` bridge
|
|
|
- to libc: read/write/open/close/unlink/rename/getcwd/chdir/opendir/
|
|
|
- readdir/closedir/statvfs + termios (tcgetattr/tcsetattr/cfmakeraw).
|
|
|
-- `shell/Shell.mod` — TP3.0 main-menu shell, same keys and layout as the
|
|
|
- original disassembled source (TPSRC4 kmenu / kcmdtab).
|
|
|
-- `shell/Makefile` — builds via `make`. Note: gm2 only emits `main` when
|
|
|
- the program module source is on the link line
|
|
|
- (`gm2 -fiso -o tpshell Shell.mod Term.o Posix.o`).
|
|
|
-- `RESUME-TP3.md` — full book-derived reference on the 8086 code the TP3
|
|
|
- compiler generates for each Pascal construct (source of the compiler
|
|
|
- phase design).
|
|
|
-- `Resources/` — TCAT pdfs / OCR of "Au coeur de Turbo Pascal",
|
|
|
- TP3.0 reference manual, disassembled TPSRC1-10.
|
|
|
-
|
|
|
-### Shell features (verified)
|
|
|
-- Exact TP3.0 screen: `Logged drive`, `Active directory`, `Work file`,
|
|
|
- `Main file`, `Edit Compile Run Save` / `Dir Quit compiler Options`,
|
|
|
- `Text: n bytes`, `Free: n bytes`, `>` prompt. Command letters drawn
|
|
|
- bold (TP3 uses high-bit chars).
|
|
|
-- Keys: L A W M E C R S D O Q; any other key redraws the menu (TP3).
|
|
|
-- W — set work file, auto-`.PAS`, `Loading`/`New File`, append-on-save.
|
|
|
-- S — save with `^Z` EOF; old file -> `.BAK` via unlink+rename (TP3 order).
|
|
|
-- D — directory listing, DOS-style `*.*` mask glob, `No files`,
|
|
|
- `k bytes free` (statvfs).
|
|
|
-- O — options submenu (M/C/H/O/D/I/A/P/F/Q) with on-screen state
|
|
|
- (compile target, Code/Data/Stack, command-line params).
|
|
|
-- Q — confirmed quit (prompts to save when text changed).
|
|
|
-- E / C / R — placeholders (editor, compiler, run to come).
|
|
|
-
|
|
|
-### Reflection notes / quirks
|
|
|
-- Model cannot view images (manual figures), so layout was reconstructed
|
|
|
- from the disassembled TPSRC source, which is authoritative for TP3.0.
|
|
|
-- Foreign modules reject Modula-2-only types (`POINTER TO record`) —
|
|
|
- use `ADDRESS` and cast at the call site.
|
|
|
-- gm2 ICEs on `FOR ... EXIT` loops (use `WHILE`+flag).
|
|
|
-- termios is used in "raw" mode; output-affected: every newline is
|
|
|
- explicit `CR LF`.
|
|
|
-- Directory glob implements DOS semantics: `*.*` matches everything,
|
|
|
- `*` matches runs, case-insensitive; skips `.`/`..`.
|
|
|
+Recreating **Turbo Pascal 3.0** (shell / editor / compiler / interpreter) in
|
|
|
+**GNU Modula-2** (`gm2 -fiso -Wall`). Faithful to the original: screen
|
|
|
+layout, keys and behaviour are reconstructed from the disassembled TP3.0
|
|
|
+source in `Resources/turbopascal3source/TP3/` (TPSRC1-10) and the reference
|
|
|
+manual, not guessed.
|
|
|
+
|
|
|
+## Milestones
|
|
|
+
|
|
|
+| Milestone | Tag | State |
|
|
|
+|---|---|---|
|
|
|
+| TP3.0 main-menu shell | `v0.1-shell` | done |
|
|
|
+| WordStar-style editor | `v0.2-editor` | done |
|
|
|
+| Shell/editor polish, Ctrl-K-D / Ctrl-K-X quit | `v-TP3-SHELL-EDITOR-QUIT` | done |
|
|
|
+| Compiler skeleton + build recipe | `v-TP3-SHELL-COMPILES` | done |
|
|
|
+| Parser `Skip` bug class (9 sites) | `v-TP3-PARSER-FIXES` | done |
|
|
|
+| Standard procedures + `rel16` fix | `v-TP3-STDPROCS` | done |
|
|
|
+| Runtime blob, linker, `CmdRun` interpreter | — | **not started** |
|
|
|
+
|
|
|
+## Build
|
|
|
+
|
|
|
+```sh
|
|
|
+cd shell && make # → shell/tpshell
|
|
|
+```
|
|
|
+
|
|
|
+Whole-program link **must** be two-phase; a single `gm2 -o` pass 3 silently
|
|
|
+caps identifier/error counts on a compiler-sized program:
|
|
|
+
|
|
|
+```sh
|
|
|
+gm2 -fiso -c Compiler.mod # phase A
|
|
|
+gm2 -fiso -fgen-module-list=modules.lst -o /dev/null \
|
|
|
+ Compiler.mod Term.o TextBuf.o Posix.o Editor.o # phase B1 (rc=1 expected)
|
|
|
+gm2 -fiso -fuse-module-list=modules.lst -o tpshell \
|
|
|
+ Shell.mod Compiler.mod Term.o TextBuf.o Posix.o Editor.o # phase B2
|
|
|
+```
|
|
|
+
|
|
|
+Current clean build: `make clean && make` → rc=0, `tpshell` **120792 bytes**.
|
|
|
+The one diagnostic is `./Compiler.mod: ParseExpr: too many errors in pass 3`,
|
|
|
+which is the expected phase-1 rollup that the recipe tolerates — not a real
|
|
|
+error. `shell/build_tpshell.sh` and `shell/Makefile` are authoritative.
|
|
|
+
|
|
|
+## What is verified, and how
|
|
|
+
|
|
|
+Nothing here is "it compiles clean" — each claim below comes from a run.
|
|
|
+
|
|
|
+### Compiler front end — `shell/tests/run_compile_tests.sh`
|
|
|
+
|
|
|
+`tests/CompileTest.mod` links `Compiler` + `TextBuf` + `Posix`, reads fixture
|
|
|
+paths from **stdin**, loads each exactly the way `LoadWorkFile` does
|
|
|
+(LF→CR normalisation, `^Z` ends the text), calls `Compile`, and prints a
|
|
|
+verdict plus a source excerpt with a caret at `errPos`. No pty, instant.
|
|
|
+
|
|
|
+```
|
|
|
+cd shell && tests/run_compile_tests.sh # all fixtures
|
|
|
+cd shell && tests/run_compile_tests.sh /some/dir # another fixture set
|
|
|
+printf '@dump\ntests/fixtures/t19_int1.pas\n' | ./compiletest # hex-dump the image
|
|
|
+```
|
|
|
+
|
|
|
+**17 of 23 fixtures compile**, up from 1 (the empty program) when the
|
|
|
+direct harness was first built.
|
|
|
+
|
|
|
+Compiling: `t01` minimal · `t04` var+assign+`writeln` · `t06` two args ·
|
|
|
+`t07` 10 assignments · `t08` const · `t09` if/then/else · `t10` while ·
|
|
|
+`t11` for/to · `t12` repeat/until · `t13` procedure + value param ·
|
|
|
+`t15` label + goto · `t16` `writeln('a')` · `t18` bare `writeln` ·
|
|
|
+`t19` `writeln(1)` · `t20` 3 string args · `t21` mixed args ·
|
|
|
+`t22` `case` with two labels.
|
|
|
+
|
|
|
+Failing, all deliberately: `t02/t03/t05/t17` multi-char string literals and
|
|
|
+`t14` `array [1..5] of integer` → `ENoLib` (102), the original's
|
|
|
+"not implemented" path; `uierror` is a deliberate syntax error used by the
|
|
|
+UI test.
|
|
|
+
|
|
|
+### Shell + editor UI — `shell/tests/uitest.py`
|
|
|
+
|
|
|
+Drives a real pty (`tests/ptyharness.py` supplies read-until-quiet, key
|
|
|
+sending and a small VT100 emulator) and asserts the TP3 **compile-error
|
|
|
+jump**: `W` load → `C` compile → `ESC` → editor opens with the cursor on the
|
|
|
+error position → `Ctrl-K D` back to the menu → `Q` exit 0. **10/10 pass**,
|
|
|
+and the reported error number is 41 (not 0 — this is what caught the
|
|
|
+`errNo := errNo` self-assignment where `Compile`'s formal shadowed the
|
|
|
+module variable).
|
|
|
+
|
|
|
+The jump mirrors original TP3 `kcwait` + `editor2` (TPSRC5:333-336, :919):
|
|
|
+`BX:=txerrpos; DEC BX; JMP editor2`, and `editor2` does `ADD BX,txbeg;
|
|
|
+INC BX` — the `DEC`/`INC` cancel, so the net is `txbeg+txerrpos` = our 0-based
|
|
|
+`errPos`. Armed as a sticky position (`Editor.GotoOffset`) rather than by
|
|
|
+changing `Run`'s signature, so `Editor.def` stays additive.
|
|
|
+
|
|
|
+`uitest.py` uses a fixture with a deliberate **syntax** error
|
|
|
+(`x := 1 + ;` → error 41, line 9 col 12) rather than a missing library
|
|
|
+feature: the latter move as the compiler grows, and a test whose
|
|
|
+expectations drift with it stops being a test.
|
|
|
+
|
|
|
+## Components
|
|
|
+
|
|
|
+### Shell — `shell/Shell.mod`, `Term.mod`, `Posix.c`, `TextBuf.mod`
|
|
|
+
|
|
|
+Exact TP3.0 screen (`Logged drive`, `Active directory`, `Work file`,
|
|
|
+`Main file`, `Edit Compile Run Save` / `Dir Quit compiler Options`,
|
|
|
+`Text: n bytes`, `Free: n bytes`, `>`), command letters drawn bold. Keys
|
|
|
+`L A W M E C R S D O Q`; any other key redraws the menu, like TP3. `W`
|
|
|
+auto-`.PAS` with `Loading`/`New File`; `S` writes `^Z` EOF and rotates the
|
|
|
+old file to `.BAK` (unlink+rename, TP3's order); `D` is a DOS `*.*` glob
|
|
|
+listing with `k bytes free`; `O` is the options submenu; `Q` confirms and
|
|
|
+prompts to save when the text changed.
|
|
|
+
|
|
|
+`E` and `C` are wired. **`R` is a stub** — it prints "Interpreter pending".
|
|
|
+
|
|
|
+### Editor — `shell/Editor.mod`
|
|
|
+
|
|
|
+WordStar-style full-screen editing over `TextBuf`. Status line
|
|
|
+`Line n Col n Insert/Overwrite Indent X:FILENAME`; text on rows 2-24.
|
|
|
+
|
|
|
+Movement (`^S/^D/^E/^X/^A/^F/^R/^C/^W/^Z`, `^Q S/D/E/X/R/C/B/K/P`, arrows,
|
|
|
+PgUp/PgDn/Home/End), editing (`^V` insert/overtype, `^G` delete char,
|
|
|
+backspace joins lines, `^T`/`^Y` word/line, `^Q-Y` to EOL, `^N`/CR break,
|
|
|
+TAB auto-indent to the word start above), block (`^K B/T/H` mark, word,
|
|
|
+show, `^K C/V/Y` copy/move/delete, `^K R/W` file read/write), search
|
|
|
+(`^Q-F`, `^Q-A`, `^L` repeat; options B/G/n/U/W, N = no-confirm; `^A` any
|
|
|
+char, CR LF matches a line break), `^P` literal control char,
|
|
|
+`^U`/ESC aborts a prompt. `^K-D` returns to the shell with the text still in
|
|
|
+memory and `changed` reported. Disk format matches TP3: CRLF plus trailing
|
|
|
+`^Z`, load normalises, save re-expands.
|
|
|
+
|
|
|
+Detail: `TP3-EDITOR.md`, `TP3-EDITOR-PSEUDOCODE.md`.
|
|
|
+
|
|
|
+### Compiler — `shell/Compiler.mod`
|
|
|
+
|
|
|
+Single-pass Pascal → 8086, following TPSRC6 `turbo` / TPSRC7-10: one pass
|
|
|
+over the shared `TextBuf` emitting machine code into `cbuf`, a patch list
|
|
|
+for forward references, TP3-style error reporting (number + relative
|
|
|
+position), and code/data size accounting. Emitted image is a byte array
|
|
|
+(mode word, CS/DS, size words, `CALL initmem`, `MOV BP,SP`, generated code).
|
|
|
+
|
|
|
+Working subset (v0.4): integer/char/boolean/byte scalars, constants with
|
|
|
+folding, globals, locals, value parameters, procedures and scalar-result
|
|
|
+functions, `ARRAY[const..const]` with constant indexing, control flow,
|
|
|
+`GOTO`/`EXIT`, and the standard procedures `WRITE`, `WRITELN`, `READ`,
|
|
|
+`READLN`, `HALT`.
|
|
|
+
|
|
|
+**Standard procedures** are `KBuiltin`, not `KProc`, because they are not
|
|
|
+called generically. TP3 (TPSRC8 `pwriteln`/`pwrloop`/`prdtyped`) does *not*
|
|
|
+pass a descriptor to the runtime: it inspects each argument's class and emits
|
|
|
+a *different call per type*, so formatting is fixed at compile time and the
|
|
|
+runtime only ever sees a value. `IoCall` mirrors that — one call per
|
|
|
+argument, then a final call for the line break:
|
|
|
+
|
|
|
+```
|
|
|
+writeln(1) MOV AX,1 ; PUSH AX ; CALL 20H ; ADD SP,2 ; CALL 40H
|
|
|
+writeln('a') MOV AX,'a' ; PUSH AX ; CALL 28H ; ADD SP,2 ; CALL 40H
|
|
|
+readln(x) LEA AX,[0104]; PUSH AX ; CALL 48H ; ADD SP,2 ; CALL 60H
|
|
|
+```
|
|
|
+
|
|
|
+`READ`/`READLN` push the *address* so the runtime can store
|
|
|
+(`EmPushVarAddr`: `8D 46 disp` / `8D 06 off`); a non-variable argument is
|
|
|
+`ETypeErr` (56), as in TP3. `TU_WrInt/Char/Bool/Real`, `TU_WrLn`,
|
|
|
+`TU_RdInt/Char/Bool`, `TU_RdLn`, `TU_Halt` continue the existing `TU_*`
|
|
|
+image-base space (`TU_InitMem=8H`, `TU_ProgEnd=10H`, `TU_StackChk=18H`).
|
|
|
+
|
|
|
+Detail: `TP3-COMPILER.md`.
|
|
|
+
|
|
|
+## Honest limitations
|
|
|
+
|
|
|
+- **A compiled image still cannot be executed.** `CmdRun` is a stub and the
|
|
|
+ runtime blob does not exist. The emitted code is verified *byte by byte*
|
|
|
+ against the offsets the compiler intends, but nothing has ever run it.
|
|
|
+- **The `TU_*` entry offsets are placeholders under an assumed model** — a
|
|
|
+ runtime blob prepended to the image, with every `pc` rebased by its size.
|
|
|
+ That model is assumed, not implemented; the linker is unwritten. Note the
|
|
|
+ prologue's `CALL TU_InitMem` targets offset 8, which is currently the
|
|
|
+ `hdrMax` header word, so the scheme is only coherent once the runtime is
|
|
|
+ actually prepended.
|
|
|
+- **No string runtime.** `RdConst` gives a 1-character literal as `TScalar`
|
|
|
+ (its char code) and only longer literals as `TString`, so multi-char
|
|
|
+ literals raise `ENoLib`. `writeln('a')` works via a `chr` flag on `ERes`;
|
|
|
+ without it the compiler emitted the *integer* writer and would have printed
|
|
|
+ 97 while the test still said OK.
|
|
|
+- **Not implemented** (all `ENoLib`): real, set, record, file, string, and
|
|
|
+ any type wider than 2 bytes. `with` is `ENoLib`. `case` *is* implemented
|
|
|
+ (cascade `CMP`/`JNZ` per label, per RESUME-TP3.md §3.6) but only over
|
|
|
+ scalar labels — subrange labels and label lists are untested.
|
|
|
+- gm2 string-literal → `ARRAY OF CHAR` assignment copies the literal *plus a
|
|
|
+ NUL* and leaves the tail untouched, so NUL-terminated tables are safe —
|
|
|
+ this was checked, not assumed.
|
|
|
+
|
|
|
+## Three bugs only a byte-level dump could find
|
|
|
+
|
|
|
+None of these produced a compile error, and two produced *passing* tests.
|
|
|
+All three were found by hex-dumping the emitted image and decoding it by
|
|
|
+hand — code sizes looked perfectly plausible throughout.
|
|
|
+
|
|
|
+1. **`rel16` off by 2 in every direct CALL/JMP.** `EmCall`/`EmJmpNear`/
|
|
|
+ `EmJcc` computed the displacement from `pc` at a point where `pc` already
|
|
|
+ pointed past the opcode and *at* the displacement field; x86 measures
|
|
|
+ from the end of the instruction (`pc + 2`). `ResolvePatches`, on the
|
|
|
+ forward-patched path, was already right — which is why forward gotos
|
|
|
+ looked fine and backward ones did not.
|
|
|
+
|
|
|
+2. **`EmMovAxSp` emitted `8B 04` with no SIB byte.** ModRM `04` means "a SIB
|
|
|
+ byte follows", so the `CMP AX,imm16` of the *next* instruction was eaten
|
|
|
+ as that SIB byte, and the intended `MOV AX,[SP]` became
|
|
|
+ `MOV AX,[BP+DI+disp]`. Every `case` label comparison therefore vanished
|
|
|
+ and `case` compiled to a chain of loads from garbage addresses — while
|
|
|
+ the fixture reported OK. Correct encoding is `8B 44 24 00` (`[SP]` cannot
|
|
|
+ use mod=00, that computes `BP+SP`).
|
|
|
+
|
|
|
+3. **The dump tool's own offset column was wrong.** It printed 4 *nibbles*
|
|
|
+ through a helper that formats a **byte**, so every row was labelled 16×
|
|
|
+ too large (`0010` shown as `00000100`). A misleading tool is worse than
|
|
|
+ none — it corrupts any offset arithmetic done from its output.
|
|
|
+
|
|
|
+## gm2 / ISO Modula-2 pitfalls hit along the way
|
|
|
+
|
|
|
+- **Two-phase link** (above) — a single whole-program pass 3 caps
|
|
|
+ identifiers/errors silently.
|
|
|
+- `EXIT` inside the program-header parameter `WHILE` **ICEs gm2** in pass 3
|
|
|
+ (`ExitStatement → PopExit → M2StackWord_PopWord → invalidloc`).
|
|
|
+ Extracting the loop into its own procedure did *not* help. Use a `BOOLEAN`
|
|
|
+ "advanced" flag, never `EXIT`. (`LOOP`+`EXIT` is fine, and is used widely.)
|
|
|
+- A bare `HALT` aborts under `-fiso` (SIGABRT, exit 134); `HALT (0)` is
|
|
|
+ correct.
|
|
|
+- `CHAR` is not the ZType: `ch = 09H` must be `ORD (ch) = 09H`.
|
|
|
+- ISO forbids dropping a function result in a statement → the `DropCh`/
|
|
|
+ `DropB`/`DropC` discard helpers wrap ~39 call sites.
|
|
|
+- `AND`/`OR`/`NOT` on 16-bit `CARDINAL` → `BitAnd`/`BitOr`/`BitNot`.
|
|
|
+- `PROCEDURE f : T` is invalid; the file's convention is
|
|
|
+ `PROCEDURE f () : T`.
|
|
|
+- ISO will not index a plain string constant as an array (needs an array
|
|
|
+ constructor) — compute hex digits with `CHR` instead.
|
|
|
+- Foreign modules (`FOR "C"`) must use `ADDRESS`, not Modula-2 pointer
|
|
|
+ types; cast at the call site. `Posix` has no `Posix.mod` (do not try to
|
|
|
+ rebuild it) and no `argv` binding.
|
|
|
+- termios raw mode: every newline is an explicit `CR LF`.
|
|
|
+- Foreign modules and `Posix` cannot pass `argv`; the test harness reads
|
|
|
+ fixture paths from **stdin** instead.
|
|
|
+
|
|
|
+## Reference material
|
|
|
+
|
|
|
+- `TP3-COMPILER.md` — the compiler: what was changed, the `Skip` bug class,
|
|
|
+ the `rel16` off-by-2, standard procedures, current matrix.
|
|
|
+- `TP3-EDITOR.md` / `TP3-EDITOR-PSEUDOCODE.md` — the original TP3 editor,
|
|
|
+ from TPSRC5/TPSRC6.
|
|
|
+- `RESUME-TP3.md` — book-derived reference for the 8086 code TP3 generates
|
|
|
+ per Pascal construct (types, skeletons, arithmetic, IF/CASE/REPEAT/WHILE/
|
|
|
+ FOR, procedures, parameters, functions, I/O, typed constants, absolutes),
|
|
|
+ plus the `TU_*` runtime entry list.
|
|
|
+- `Resources/turbopascal3source/TP3/` — TPSRC1-10, the disassembled original.
|
|
|
+ **This is the ground truth**; when our behaviour and the book disagree, the
|
|
|
+ disassembly wins.
|
|
|
+- `Resources/coeur-tp-ocr/` — OCR of "Au coeur de Turbo Pascal".
|
|
|
|
|
|
## Next steps
|
|
|
-1. Editor: WordStar-style keys (^A..^Z basic editing, arrows, insert/overtype),
|
|
|
- reusing the `txt` text buffer already in Shell.mod.
|
|
|
-2. Compiler: front-end + 8086 code generator per RESUME-TP3.md
|
|
|
- (program prologue with CS-prefixed typed constants, TU_STACK_CHECK
|
|
|
- parameter pushes, VAR parameter les/seg access, functions in AX /
|
|
|
- TU_RESU_FCT_ST strings, typed constants in code segment, absolute vars).
|
|
|
-3. Interpreter ("R") for TP3 .COM-compatible execution.
|
|
|
+
|
|
|
+1. **Runtime blob + linker.** Everything downstream is gated on it: the
|
|
|
+ `TU_*` entries have to land somewhere real and every `pc` needs rebasing.
|
|
|
+2. **`CmdRun`** — the interpreter. Without it no compiled program has ever
|
|
|
+ run, so the codegen is unproven in the only way that counts.
|
|
|
+3. **String runtime** — unlocks the last 4 real fixture failures.
|
|
|
+4. Nested procedures / recursion, `var` parameters (the `SEG:OFF` push from
|
|
|
+ RESUME-TP3.md §3.11), range/index checks (`TU_RANGE_CHECK`,
|
|
|
+ `TU_INDEX_CHECK`), and typed constants (RESUME-TP3.md §3.14).
|
