Home Verkennen Help
Inloggen
eric
/
TP3
1
0
Vork 0
Bestanden Issues 0 Pull-aanvragen 0 Wiki
Boom: 2bef202bdd
Aftakkingen Labels
TP3
/
SUMMARY.md

SUMMARY.md 22 KB
Geschiedenis Ruwe

TP3-comp — state summary

Recreating Turbo Pascal 3.0 (shell / editor / compiler / interpreter) in GNU Modula-2 (gm2 -fiso -Wall). Faithful to the original: screen layout, keys and behaviour are reconstructed from the disassembled TP3.0 source in Resources/turbopascal3source/TP3/ (TPSRC1-10) and the reference manual, not guessed.

Milestones

Milestone Tag State
TP3.0 main-menu shell v0.1-shell done
WordStar-style editor v0.2-editor done
Shell/editor polish, Ctrl-K-D / Ctrl-K-X quit v-TP3-SHELL-EDITOR-QUIT done
Compiler skeleton + build recipe v-TP3-SHELL-COMPILES done
Parser Skip bug class (9 sites) v-TP3-PARSER-FIXES done
Standard procedures + rel16 fix v-TP3-STDPROCS done
Runtime library + 8086 execution harness v-TP3-RUNTIME-BLOB assembled, never run
Linker + CmdRun — not started

Build

cd shell && make          # → shell/tpshell

Whole-program link must be two-phase; a single gm2 -o pass 3 silently caps identifier/error counts on a compiler-sized program:

gm2 -fiso -c Compiler.mod                                     # phase A
gm2 -fiso -fgen-module-list=modules.lst -o /dev/null \
        Compiler.mod Term.o TextBuf.o Posix.o Editor.o        # phase B1 (rc=1 expected)
gm2 -fiso -fuse-module-list=modules.lst -o tpshell \
        Shell.mod Compiler.mod Term.o TextBuf.o Posix.o Editor.o   # phase B2

Current clean build: make clean && make → rc=0, tpshell 120792 bytes. The one diagnostic is ./Compiler.mod: ParseExpr: too many errors in pass 3, which is the expected phase-1 rollup that the recipe tolerates — not a real error. shell/build_tpshell.sh and shell/Makefile are authoritative.

What is verified, and how

Nothing here is "it compiles clean" — each claim below comes from a run.

Compiler front end — shell/tests/run_compile_tests.sh

tests/CompileTest.mod links Compiler + TextBuf + Posix, reads fixture paths from stdin, loads each exactly the way LoadWorkFile does (LF→CR normalisation, ^Z ends the text), calls Compile, and prints a verdict plus a source excerpt with a caret at errPos. No pty, instant.

cd shell && tests/run_compile_tests.sh            # all fixtures
cd shell && tests/run_compile_tests.sh /some/dir  # another fixture set
printf '@dump\ntests/fixtures/t19_int1.pas\n' | ./compiletest   # hex-dump the image

17 of 23 fixtures compile, up from 1 (the empty program) when the direct harness was first built.

Compiling: t01 minimal · t04 var+assign+writeln · t06 two args · t07 10 assignments · t08 const · t09 if/then/else · t10 while · t11 for/to · t12 repeat/until · t13 procedure + value param · t15 label + goto · t16 writeln('a') · t18 bare writeln · t19 writeln(1) · t20 3 string args · t21 mixed args · t22 case with two labels.

Failing, all deliberately: t02/t03/t05/t17 multi-char string literals and t14 array [1..5] of integer → ENoLib (102), the original's "not implemented" path; uierror is a deliberate syntax error used by the UI test.

Shell + editor UI — shell/tests/uitest.py

Drives a real pty (tests/ptyharness.py supplies read-until-quiet, key sending and a small VT100 emulator) and asserts the TP3 compile-error jump: W load → C compile → ESC → editor opens with the cursor on the error position → Ctrl-K D back to the menu → Q exit 0. 10/10 pass, and the reported error number is 41 (not 0 — this is what caught the errNo := errNo self-assignment where Compile's formal shadowed the module variable).

The jump mirrors original TP3 kcwait + editor2 (TPSRC5:333-336, :919): BX:=txerrpos; DEC BX; JMP editor2, and editor2 does ADD BX,txbeg; INC BX — the DEC/INC cancel, so the net is txbeg+txerrpos = our 0-based errPos. Armed as a sticky position (Editor.GotoOffset) rather than by changing Run's signature, so Editor.def stays additive.

uitest.py uses a fixture with a deliberate syntax error (x := 1 + ; → error 41, line 9 col 12) rather than a missing library feature: the latter move as the compiler grows, and a test whose expectations drift with it stops being a test.

The executor problem (blocking everything downstream)

The whole point of a Pascal→8086 compiler is that the output runs. Until this milestone no compiled image had ever been executed, so the plan was: get a real CPU emulator, run the image, assert the exact stdout bytes.

Unicorn 2.1.4 cannot be used for this. UC_MODE_16 mis-decodes 16-bit ModRM memory operands. The measurement, by loading a byte-pattern image so a load reveals its own effective address, then executing a single LEA AX,[r+disp8] and reading AX back with every register set to a distinct value:

mod=01, disp8=4            got        8086 says
  8D 40 04  LEA AX,[BX+4]  AX=0d04    0x504   (= BX+SI+4)   WRONG
  8D 41 04  LEA AX,[BX+SI+4] AX=0e04  0xd04                WRONG
  8D 42 04  LEA AX,[BX+DI+4] AX=0f04  0xe04                WRONG
  8D 43 04  LEA AX,[BP+4]  AX=1004    0x704                WRONG
  8D 45 04  LEA AX,[DI+4]  AX=0904    0x904                right
  8D 46 04  LEA AX,[BP+4]  AX=0704    0x704                right
  8D 47 04  LEA AX,[DI+4]  AX=0504    0x904                WRONG
mod=00 / mod=10 direct disp16
  8B 1E 00 20  MOV BX,[2000]  BX=1234                     right
  8D 06 34 12  LEA AX,[1234]  AX=1234                     right

So rm=5 and rm=6 decode correctly but rm=0,1,2,3,7 do not, and only base-register-free addressing (direct disp16) is trustworthy. That rules Unicorn out as an oracle for exactly the instruction forms generated code is made of — LEA AX,[BP+d], MOV AX,[SI+d], LODSW-style loops, everything with a frame pointer. It cannot distinguish "my codegen is wrong" from "the emulator is wrong", which makes it worse than no emulator at all.

pip install --upgrade unicorn resolves to the same 2.1.4, so this is not avoidable by upgrading.

Also ruled out, for the record:

  • DOSBox-X 2025.02.01 (installed, /usr/bin/dosbox-x, plain root-owned ELF, not a snap) starts cleanly headless under SDL_VIDEODRIVER=dummy SDL_AUDIODRIVER=dummy, but its -c/autoexec commands never observably execute: a md never appeared on the host, and a .COM that creates OUT.TXT via INT 21h AH=3Dh/40h never produced the file. Shell > is intercepted by dosbox-x's own wrapper (SHELL:Redirect output to out.txt). A .BAT route timed out.
  • qemu-system-i386 is installed (/usr/bin/qemu-system-i386) and is the next candidate: a 512-byte boot sector can load the image and a INT 21h shim can hand the output back. Not attempted yet.

tests/rt_exec.py is the harness, written against Unicorn, and it is written to survive the switch: it loads the runtime, calls each entry with a known argument, and compares the bytes sent to INT 21h against expectations. It currently fails, and the failures are the emulator's, not the library's — wrint prints - for every value because MOV AX,[BP+4] reads the wrong address. Do not read those results as a verdict on the runtime.

Components

Shell — shell/Shell.mod, Term.mod, Posix.c, TextBuf.mod

Exact TP3.0 screen (Logged drive, Active directory, Work file, Main file, Edit Compile Run Save / Dir Quit compiler Options, Text: n bytes, Free: n bytes, >), command letters drawn bold. Keys L A W M E C R S D O Q; any other key redraws the menu, like TP3. W auto-.PAS with Loading/New File; S writes ^Z EOF and rotates the old file to .BAK (unlink+rename, TP3's order); D is a DOS *.* glob listing with k bytes free; O is the options submenu; Q confirms and prompts to save when the text changed.

E and C are wired. R is a stub — it prints "Interpreter pending".

Editor — shell/Editor.mod

WordStar-style full-screen editing over TextBuf. Status line Line n Col n Insert/Overwrite Indent X:FILENAME; text on rows 2-24.

Movement (^S/^D/^E/^X/^A/^F/^R/^C/^W/^Z, ^Q S/D/E/X/R/C/B/K/P, arrows, PgUp/PgDn/Home/End), editing (^V insert/overtype, ^G delete char, backspace joins lines, ^T/^Y word/line, ^Q-Y to EOL, ^N/CR break, TAB auto-indent to the word start above), block (^K B/T/H mark, word, show, ^K C/V/Y copy/move/delete, ^K R/W file read/write), search (^Q-F, ^Q-A, ^L repeat; options B/G/n/U/W, N = no-confirm; ^A any char, CR LF matches a line break), ^P literal control char, ^U/ESC aborts a prompt. ^K-D returns to the shell with the text still in memory and changed reported. Disk format matches TP3: CRLF plus trailing ^Z, load normalises, save re-expands.

Detail: TP3-EDITOR.md, TP3-EDITOR-PSEUDOCODE.md.

Compiler — shell/Compiler.mod

Single-pass Pascal → 8086, following TPSRC6 turbo / TPSRC7-10: one pass over the shared TextBuf emitting machine code into cbuf, a patch list for forward references, TP3-style error reporting (number + relative position), and code/data size accounting. Emitted image is a byte array (mode word, CS/DS, size words, CALL initmem, MOV BP,SP, generated code).

Working subset (v0.4): integer/char/boolean/byte scalars, constants with folding, globals, locals, value parameters, procedures and scalar-result functions, ARRAY[const..const] with constant indexing, control flow, GOTO/EXIT, and the standard procedures WRITE, WRITELN, READ, READLN, HALT.

Standard procedures are KBuiltin, not KProc, because they are not called generically. TP3 (TPSRC8 pwriteln/pwrloop/prdtyped) does not pass a descriptor to the runtime: it inspects each argument's class and emits a different call per type, so formatting is fixed at compile time and the runtime only ever sees a value. IoCall mirrors that — one call per argument, then a final call for the line break:

writeln(1)    MOV AX,1     ; PUSH AX ; CALL 20H ; ADD SP,2 ; CALL 40H
writeln('a')  MOV AX,'a'   ; PUSH AX ; CALL 28H ; ADD SP,2 ; CALL 40H
readln(x)     LEA AX,[0104]; PUSH AX ; CALL 48H ; ADD SP,2 ; CALL 60H

READ/READLN push the address so the runtime can store (EmPushVarAddr: 8D 46 disp / 8D 06 off); a non-variable argument is ETypeErr (56), as in TP3. TU_WrInt/Char/Bool/Real, TU_WrLn, TU_RdInt/Char/Bool, TU_RdLn, TU_Halt continue the existing TU_* image-base space (TU_InitMem=8H, TU_ProgEnd=10H, TU_StackChk=18H).

Detail: TP3-COMPILER.md.

Runtime library — shell/Runtime.mod, shell/Runtime.def

The 8086 runtime, assembled byte by byte from Modula-2 — no external assembler and no checked-in binary, so the tree stays self-contained and the entry offsets are derived rather than guessed. Each emitter is one instruction with its ModRM byte spelled out in a comment so the encoding can be checked by hand against an 8086 table. This is the same approach Compiler.mod already takes (Ebyte/Eword/EmCall).

It mirrors the original's own mechanism: TPSRC7 copyrt copies the runtime into the front of the code buffer (SI=DI=0, REPZ MOVSB) and pc is then initialised past it (MOV pc,#$2D7C). Same shape here — the compiler is meant to copy the blob to the front of cbuf and start pc/dc past it. Runtime data therefore lives at fixed low offsets and needs no relocation, and because both sides of every CALL shift by the same amount, EmCall's displacement arithmetic is unaffected by the runtime being prepended.

Current blob: 366 bytes, 13 entries, offsets read back out of the assembled bytes by tests/RtProbe.mod:

entry offset entry offset entry offset
initmem 0 wrint 36 rdint 142
progend 28 wrchar 97 rdchar 243
stackchk 35 wrbool 106 rdbool 264
halt 28 wrreal 126 rdln 310
wrln 134

progend and halt deliberately share one address (XOR AX,AX / MOV AH,4C / INT 21h / RET): the compiler already zeroes AX before progend and discards the HALT argument at compile time, so both leave with exit code 0.

Conventions, matching Compiler.IoCall exactly:

entry argument notes
WrInt/WrChar/WrBool/WrReal one 16-bit value on the stack caller pops
RdInt/RdChar/RdBool one address on the stack caller pops
WrLn/RdLn/StackChk nothing
InitMem AX = offset of the program header a register, not a stack word
ProgEnd/Halt nothing exits, code 0

InitMem reads the data base and end out of the header words at +2/+6 and zeroes that range, because Pascal leaves globals undefined. StackChk is a bare RET — range and stack checking aren't compiled in yet, and the call site sits mid-expression, so it must not touch a register.

Two label-name plus fixup list: rel8, rel16 and runtime-data addresses are all patched after the blob is placed, so nothing depends on a hand-computed displacement.

It has never executed. See the emulator finding below.

Honest limitations

  • A compiled image still cannot be executed. CmdRun is a stub, the linker is unwritten, and no 8086 executor on this machine has yet proved trustworthy (above). The emitted code is verified byte by byte against the offsets the compiler intends, but nothing has ever run it.
  • The runtime is written but unproven. Runtime.mod assembles to 366 bytes and its entry offsets are derived from the emitted bytes, but it has never been executed on a correct CPU, so treat every encoding in it as unverified even though three of its own bugs were caught by running it under a broken emulator.
  • The compiler is not yet wired to the runtime. Compiler.mod still carries the hardcoded placeholder TU_* constants (TU_InitMem=8H, TU_ProgEnd=10H, …) and still starts pc at 0, so emitted images do not contain the runtime and those offsets are still wrong. Runtime.mod is not in make or run_compile_tests.sh yet for the same reason. Note the prologue's CALL TU_InitMem targets offset 8, which is currently the hdrMax header word — coherent only once the blob is really prepended. *(Correction to the earlier note in this file: the TU_InitMem=8 "collision" was a false alarm. Per TPSRC7 the runtime is copied to the front of the code buffer and pc starts past it, so TU_* offsets are runtime-relative, not image-absolute, and no rebasing of the displacement arithmetic is needed.)*
  • No string runtime. RdConst gives a 1-character literal as TScalar (its char code) and only longer literals as TString, so multi-char literals raise ENoLib. writeln('a') works via a chr flag on ERes; without it the compiler emitted the integer writer and would have printed 97 while the test still said OK.
  • Not implemented (all ENoLib): real, set, record, file, string, and any type wider than 2 bytes. with is ENoLib. case is implemented (cascade CMP/JNZ per label, per RESUME-TP3.md §3.6) but only over scalar labels — subrange labels and label lists are untested.
  • gm2 string-literal → ARRAY OF CHAR assignment copies the literal plus a NUL and leaves the tail untouched, so NUL-terminated tables are safe — this was checked, not assumed.

Three bugs only a byte-level dump could find

None of these produced a compile error, and two produced passing tests. All three were found by hex-dumping the emitted image and decoding it by hand — code sizes looked perfectly plausible throughout.

  1. rel16 off by 2 in every direct CALL/JMP. EmCall/EmJmpNear/ EmJcc computed the displacement from pc at a point where pc already pointed past the opcode and at the displacement field; x86 measures from the end of the instruction (pc + 2). ResolvePatches, on the forward-patched path, was already right — which is why forward gotos looked fine and backward ones did not.

  2. EmMovAxSp emitted 8B 04 with no SIB byte. ModRM 04 means "a SIB byte follows", so the CMP AX,imm16 of the next instruction was eaten as that SIB byte, and the intended MOV AX,[SP] became MOV AX,[BP+DI+disp]. Every case label comparison therefore vanished and case compiled to a chain of loads from garbage addresses — while the fixture reported OK. Correct encoding is 8B 44 24 00 ([SP] cannot use mod=00, that computes BP+SP).

  3. The dump tool's own offset column was wrong. It printed 4 nibbles through a helper that formats a byte, so every row was labelled 16× too large (0010 shown as 00000100). A misleading tool is worse than none — it corrupts any offset arithmetic done from its output.

Bugs found by actually running code

Executing the hand-assembled runtime — even under a broken emulator — paid for itself immediately, because a wrong encoding executes rather than failing to assemble. Four, none of which a compiler diagnostic would ever have reported.

  1. B() silently truncated multi-byte opcodes. PROCEDURE B emits exactly one byte and masks with MOD 100H, so B (8BE4H) — a two-byte opcode passed as one literal — emitted just E4. MOV BP,SP was missing from every frame in the runtime, so BP stayed 0 and every BP-relative access read address 4. Found by decoding the hex dump; the byte is gone, not wrong.
  2. MOV BP,SP encoded as 8B E4, which is MOV SP,SP — a no-op. The ModRM byte is mod·64 + reg·8 + rm, and I mis-derived it. Compiler.mod's EmMovBpSp had it right all along (8B 0CH); only the new module was wrong. This is why the encoding is now written as three explicit B calls with the arithmetic in a comment rather than as one hex literal.
  3. InitMem read its argument from [SP] — the return address. The convention is a register (AX), unlike the per-argument I/O entries which do take a stack word. Caught because the data area was never cleared.
  4. EmPushVarAddr computes the wrong base register for locals (pre-existing, not yet fixed). It emits 8D 46 disp, which is LEA AX,[SI+disp8], but intends LEA AX,[BP+disp8] = 8D 45 disp. So read into a local variable has always addressed the wrong cell, silently. It also truncates the offset with off MOD 100H, losing displacements above 255. The global form 8D 06 off (LEA AX,[disp16]) is correct. Found while writing the runtime's read entries, which needed the same encoding to be right.

gm2 / ISO Modula-2 pitfalls hit along the way

  • Two-phase link (above) — a single whole-program pass 3 caps identifiers/errors silently.
  • EXIT inside the program-header parameter WHILE ICEs gm2 in pass 3 (ExitStatement → PopExit → M2StackWord_PopWord → invalidloc). Extracting the loop into its own procedure did not help. Use a BOOLEAN "advanced" flag, never EXIT. (LOOP+EXIT is fine, and is used widely.)
  • A bare HALT aborts under -fiso (SIGABRT, exit 134); HALT (0) is correct.
  • CHAR is not the ZType: ch = 09H must be ORD (ch) = 09H.
  • gm2's ISO SYSTEM exports ORD but not Ord — the import is case-sensitive here despite gm2's usual case-insensitivity, so FROM SYSTEM IMPORT Ord fails with "unknown symbol" while plain ORD (c) compiles. Write ORD, never Ord.
  • ISO forbids dropping a function result in a statement → the DropCh/ DropB/DropC discard helpers wrap ~39 call sites.
  • AND/OR/NOT on 16-bit CARDINAL → BitAnd/BitOr/BitNot.
  • PROCEDURE f : T is invalid; the file's convention is PROCEDURE f () : T.
  • ISO will not index a plain string constant as an array (needs an array constructor) — compute hex digits with CHR instead.
  • Foreign modules (FOR "C") must use ADDRESS, not Modula-2 pointer types; cast at the call site. Posix has no Posix.mod (do not try to rebuild it) and no argv binding.
  • termios raw mode: every newline is an explicit CR LF.
  • Foreign modules and Posix cannot pass argv; the test harness reads fixture paths from stdin instead.

Reference material

  • TP3-COMPILER.md — the compiler: what was changed, the Skip bug class, the rel16 off-by-2, standard procedures, current matrix.
  • TP3-EDITOR.md / TP3-EDITOR-PSEUDOCODE.md — the original TP3 editor, from TPSRC5/TPSRC6.
  • RESUME-TP3.md — book-derived reference for the 8086 code TP3 generates per Pascal construct (types, skeletons, arithmetic, IF/CASE/REPEAT/WHILE/ FOR, procedures, parameters, functions, I/O, typed constants, absolutes), plus the TU_* runtime entry list.
  • Resources/turbopascal3source/TP3/ — TPSRC1-10, the disassembled original. This is the ground truth; when our behaviour and the book disagree, the disassembly wins.
  • Resources/coeur-tp-ocr/ — OCR of "Au coeur de Turbo Pascal".

Next steps

  1. Get a trustworthy 8086 executor, because it gates items 2–4 and nothing else can be claimed until a compiled image runs. qemu-system-i386 with a boot-sector loader and an INT 21h shim is the plan; keep rt_exec.py's expectations and change only the machine behind them. (Do not reach for Unicorn's 16-bit mode again, and do not re-derive the rm table by hand again — check it against a real decoder.)
  2. Fix EmPushVarAddr — 8D 45/8D 85 for locals, full disp16, per bug 4 above. Two lines, and read into a local is wrong until it is done.
  3. Wire the runtime in and write the linker: pc := RT_Size, dc := RT_Size + 1000H (a fixed 4 KiB code/data gap, so a program's data can't collide with its code in a single 64 K .COM segment), take the TU_* offsets from RT_Entry instead of the hardcoded constants, patch the header words (hdrDS = data base, hdrHeap = data end, so InitMem can zero globals), pad the image to cover the data area, and emit the .COM. Add Runtime to the make and run_compile_tests.sh rebuild lists.
  4. Prove it end to end: compile a fixture, link, execute, assert the exact stdout bytes (writeln('hi') → hi). That single assertion is what turns this from "assembles" into "works".
  5. CmdRun — run the emitted image from the R menu key.
  6. String runtime — unlocks the last 4 real fixture failures.
  7. Nested procedures / recursion, var parameters (the SEG:OFF push from RESUME-TP3.md §3.11), range/index checks (TU_RANGE_CHECK, TU_INDEX_CHECK), typed constants (RESUME-TP3.md §3.14).
  8. Harden the program-header parameter loop against non-advancing input (program p(1;)) with a BOOLEAN flag — not EXIT, which ICEs gm2.