Strona główna Odkrywaj Pomoc
Zaloguj się
eric
/
TP3
1
0
Forkuj 0
Pliki Problemy 0 Oczekujące zmiany 0 Wiki
Drzewo: v-TP3-DEAD
Gałęzie Tagi
TP3
/
SUMMARY.md

SUMMARY.md 75 KB
Bezpośredni odnośnik Historia Czysty

TP3-comp — state summary

Recreating Turbo Pascal 3.0 (shell / editor / compiler / interpreter) in GNU Modula-2 (gm2 -fiso -Wall). Faithful to the original: screen layout, keys and behaviour are reconstructed from the disassembled TP3.0 source in Resources/turbopascal3source/TP3/ (TPSRC1-10) and the reference manual, not guessed.

Milestones

Milestone Tag State
TP3.0 main-menu shell v0.1-shell done
WordStar-style editor v0.2-editor done
Shell/editor polish, Ctrl-K-D / Ctrl-K-X quit v-TP3-SHELL-EDITOR-QUIT done
Compiler skeleton + build recipe v-TP3-SHELL-COMPILES done
Parser Skip bug class (9 sites) v-TP3-PARSER-FIXES done
Standard procedures + rel16 fix v-TP3-STDPROCS done
Runtime library + 8086 execution harness v-TP3-RUNTIME-BLOB assembled, executed under qemu
Inline string literals (writeln('hi')) v-TP3-STRLITERAL done, executed
Linker: real DOS .COM writer + independent byte checker v-TP3-COM-IMAGE done, executed
Measured encodings: ModR/M table, runtime audit, golden disassembly, [BP+off] v-TP3-MEASURED-EMITTERS done, executed
Execution: a boot sector, qemu, and a claim about behaviour v-TP3-EXECUTION done, 21/21 fixtures run, exact output
Execute the nine fixtures that only compiled v-TP3-DEAD-FIXTURES done, 30/30 run; four operator/scoping bugs found
Runtime entries under qemu, and the register contract stated v-TP3-BP-CONTRACT done, 36/36 entry checks; wrchar/wrbool no longer destroy BP
CmdRun (the R key), in-process 8086 interpreter — not started

Build

cd shell && make          # → shell/tpshell

Whole-program link must be two-phase; a single gm2 -o pass 3 silently caps identifier/error counts on a compiler-sized program:

gm2 -fiso -c Compiler.mod                                     # phase A
gm2 -fiso -fgen-module-list=modules.lst -o /dev/null \
        Compiler.mod Term.o TextBuf.o Posix.o Editor.o        # phase B1 (rc=1 expected)
gm2 -fiso -fuse-module-list=modules.lst -o tpshell \
        Shell.mod Compiler.mod Term.o TextBuf.o Posix.o Editor.o   # phase B2

Current clean build: make clean && make → rc=0, tpshell 163616 bytes. The one diagnostic is ./Compiler.mod: ParseExpr: too many errors in pass 3, which is the expected phase-1 rollup that the recipe tolerates — not a real error. shell/Makefile is the single authoritative build recipe.

shell/build_tpshell.sh is now a three-line wrapper around make, and that is a fix, not a refactor. It used to be a second hand-maintained copy of the recipe, and a copy of a build recipe drifts — this one was wrong twice over: it ran gm2 -c Posix.mod, but Posix is a foreign C module built by cc -c Posix.c and there is no Posix.mod, so it died on the third module every time; and it printed phase 2's return code and then carried on regardless, so a failed link was reported as a success whenever an older tpshell was still lying around. The copy that is wrong is the one nobody runs, which is how it survived. The wrapper keeps the old invocation working and holds no build knowledge of its own.

What is verified, and how

Nothing here is "it compiles clean" — each claim below comes from a run.

Compiler front end — shell/tests/run_compile_tests.sh

tests/CompileTest.mod links Compiler + TextBuf + Posix, reads fixture paths from stdin, loads each exactly the way LoadWorkFile does (LF→CR normalisation, ^Z ends the text), calls Compile, and prints a verdict plus a source excerpt with a caret at errPos. No pty, instant.

cd shell && tests/run_compile_tests.sh            # all fixtures
cd shell && tests/run_compile_tests.sh /some/dir  # another fixture set
printf '@dump \ntests/fixtures/t19_int1.pas\n' | ./compiletest  # hex-dump (trailing space! see TP3-COMPILER.md)

The matrix asserts; it does not just count. tests/fixtures/expected.tsv pins the verdict and the numbers per fixture — verdict plus code size plus data size, or error number plus position — and the runner compares. A wrong error position or a program that lost six bytes now fails the suite instead of needing a squint. It was checked for vacuousness by reverting the string scanner fix: 19/23 and exit 1, restored: 23/23 and exit 0.

30 of 33 fixtures compile clean, up from 1 (the empty program) when the direct harness was first built.

compile matrix: 33 passed, 0 failed (of 33)

Compiling: t01 minimal · t04 var+assign+writeln · t06 two args · t07 10 assignments · t08 const · t09 if/then/else · t10 while · t11 for/to · t12 repeat/until · t13 procedure + value param · t15 label + goto · t16 writeln('a') · t18 bare writeln · t19 writeln(1) · t20 3 string args · t21 mixed args · t22 case with two labels · t23 writeln('') · t24 writeln('don''t') · t26 mixed scalar/string args · t02/t03/t05/t17 multi-char literals · t27 five locals · t28 a 70-parameter declaration · t29 readln from a supplied input file · t30 a counted for whose bounds come from an expression · t31 a value parameter and a call across statements · t32 EXIT out of a for body.

Failing, all deliberately: t14 array [1..5] of integer at its point of use and t25 a string literal used as a value (s := 'hi') both → ENoLib (102), the original's "not implemented" path; uierror is a deliberate syntax error (41) used by the UI test.

run_com_tests.sh additionally links all 30 that compile to a real .COM and re-verifies the bytes with an independent Python checker that measures the layout instead of restating it: 30 checked, 0 failed. That last part was itself a bug fix — see "the two restated constants" below.

Shell + editor UI — shell/tests/uitest.py

Drives a real pty (tests/ptyharness.py supplies read-until-quiet, key sending and a small VT100 emulator) and asserts the TP3 compile-error jump: W load → C compile → ESC → editor opens with the cursor on the error position → Ctrl-K D back to the menu → Q exit 0. 10/10 pass, and the reported error number is 41 (not 0 — this is what caught the errNo := errNo self-assignment where Compile's formal shadowed the module variable).

The jump mirrors original TP3 kcwait + editor2 (TPSRC5:333-336, :919): BX:=txerrpos; DEC BX; JMP editor2, and editor2 does ADD BX,txbeg; INC BX — the DEC/INC cancel, so the net is txbeg+txerrpos = our 0-based errPos. Armed as a sticky position (Editor.GotoOffset) rather than by changing Run's signature, so Editor.def stays additive.

uitest.py uses a fixture with a deliberate syntax error (x := 1 + ; → error 41, line 9 col 12) rather than a missing library feature: the latter move as the compiler grows, and a test whose expectations drift with it stops being a test.

The encodings — six checks that can each go red

Nothing above looks at machine code. It all stops at "the compiler produced what it intended to produce", which is exactly where the bugs in this project live: a wrong ModRM byte is not a compile error and not a wrong code size, it is a perfectly well-formed instruction that does something else. So the encodings get their own stack, and it is built so that each check fails on a different class of mistake. tests/run_all.sh runs all of them; tests/nonvacuity.sh proves each one can go red.

check what it asserts what it cannot see
probe/run_modrm19.py the mod=00/01/10 effective addresses, by executing 23 cases on a real 8086 under qemu and scanning for where the marker landed mod=11 — see below
probe/modrm11.py the mod=11 register identities, by encoding with GNU as and decoding with FCML, against hard-coded bytes the table agreeing with itself
audit_helpers.py every one-line emitter in both Runtime.mod and Compiler.mod decodes to what its name says — 100/100, from an inventory scanned independently of the parser anything longer than one instruction
check_runtime.py + runtime.golden the built runtime's code region (436 bytes, code ends at 405) sweeps cleanly through FCML, every entry and all branch targets land on an instruction boundary, and the whole disassembly is byte-for-byte the committed golden whether the golden is right
check_framedisp.py [BP+off] uses disp8 iff off <= 127, for locals (negative) and far parameters (>127) which of the two encodings was chosen, if the other also works
run_com_exec.py the emitted image executes and prints exactly the expected bytes semantics the fixture never exercises
rt_exec.py the runtime entries themselves are called directly, one qemu boot per call, 35 cases plus a pre-flight, 36/36 — plus a check_bp_contract pre-flight over the built blob: an entry that borrows BP must open with 55 8B EC (exact) and contain a 5D (a screen, because 5D is also a displacement byte) whether the caller's half of a contract is right, where the only witness is a case that happens to make two calls in a row

Two of these deserve the detail, because the reason they exist is the reason they are hard.

audit_helpers.py exists because a wrong ModRM that still decodes is invisible. The mistake this project actually makes is not a malformed instruction — it is MovSiBx emitting 89 DC, which decodes perfectly as MOV SP,BX, and CmpSiBx emitting 39 DC, which decodes as CMP SP,BX. Both shipped for a long time. A structural check passes them. A golden passes them. Only asking "does this byte sequence mean what this procedure is called?" fails, which is what the audit does: it disassembles each one-line emitter and compares the decode against the name. Five bugs came out of it in one pass.

And then the audit itself turned out to have three independent ways of silently dropping subjects, which is worse, because a check that quietly checks nothing looks exactly like a check that passes:

  1. It swept Runtime.mod only — while EmXchgAxCx lives in Compiler.mod and was wrong (93h = XCHG AX,BX) for its entire life, with a correct byte count and a green matrix. The fix sweeps both modules.
  2. The procedure-header regex required a non-empty parameter list, so it matched 0 of Compiler.mod's 22 emitters.
  3. The coverage list was derived from the parser's own output. That is vacuous: any input that stops the parser also deletes the helper from the inventory, so the audit reports "100% covered" for a file it read nothing from. The inventory is now a deliberately dumb ^PROCEDURE\s+(\w+) scan with no parameter, BEGIN or comment awareness, and the audit fails if a documented emitter is missing from it. EmitData is the one named exclusion, with the reason recorded in the source.

audit_vx_helpers had a smaller version of the same disease: it printed len(VX_KINDS) = "4 Vx subjects" for Compiler.mod, which has none. It now reports the number actually checked.

modrm11.py is deliberately not self-referential. The obvious way to check a ModR/M table is to write the table in assembly and assemble it — but that can never fail, because editing the assembly makes as faithfully re-encode the new claim and the two then agree again. That failure mode was found by corrupting the .s and watching the check stay green. Two things close it: an EXPECT byte sequence hard-coded independently of the .s text, and four anchor encodings that are spelled as literal .byte directives because as would never choose them for a mnemonic (83 C4 08 ADD SP,8 · 83 C6 02 ADD SI,2 · 8B EC MOV BP,SP · 8B E5 MOV SP,BP). No table shifted by one cell can satisfy all four.

check_framedisp.py exists because the bug it guards was accidentally correct. The old code truncated the displacement with off MOD 100H, always emitting disp8. Locals are allocated downward from 0FFFEh, so a local's offset is negative and −32768..+127 — the whole range where truncating to a byte happens to be right. The bug was only visible above +127, reachable from the 63rd parameter onward, and no fixture had one. The fix is disp8 iff off <= 127 else disp16, with no overflow branch at all: disp16 covers the entire 16-bit range as a signed value. The check also asserts the rule rather than one encoding — always-disp16 is accepted, and nonvacuity.sh proves that by building it and requiring the check to stay green.

Execution under qemu — tests/run_com_exec.py

The sixth check is the one that cannot be written as a byte comparison, so it is also the one that finds the most: 30 fixtures are compiled to .COM, put on a floppy, booted, and their serial output compared to a committed .out file exactly — CRLF included — plus the exit code passed to INT 21h AH=4Ch.

execution: 30 passed, 0 failed (of 30)

The two fixtures it found nothing in are the interesting ones: t31_procparam and t32_forexit were the two most expensive bugs in the project, and neither was visible as a wrong byte count. See overProc below.

The .COM layout constants are measured, not restated. run_com_tests.sh and comtest.py both used to hard-code RT_SZ = 391 against a runtime that had since grown to 432 bytes at the time, so they read the program header 41 bytes early and reported 30 false failures — a red suite that meant nothing, which is the most expensive kind of red. Both now locate the header by its own signature (hdrFlag = 1, hdrDS == hdrOff + 1000h + bias, hdrHeap > hdrDS, hdrCS leaves room, initmem length at ENT_SZ) and derive rtSz, prologAt and dataBase per file:

measured runtime size: 436 bytes (header at image offset 439)

A restated constant that has drifted is worse than a derived one, and the two checkers had drifted from each other as well as from the runtime — which is why there are two of them and why both were wrong in the same way.

Non-vacuity — tests/nonvacuity.sh

Every assertion in this file is proved able to fail: 28 deliberate breakages, each asserted to turn exactly one named check red for the stated reason, then restored and re-asserted green. Six break the runtime, five attack the mod=11 table (including restoring the exact wrong table this project once shipped), three target EmBpDisp — the truncation, the always-disp16 over-encoding that must stay green, and the restored source — six attack the helper audit, and five attack the .COM layout checker.

Two properties of the harness itself are enforced, because both had already gone wrong silently:

  • A baseline assertion runs first, so a case that is already red is reported as NOT NON-VACUOUS and distinguished from one that went red. Without it, a harness broken by an earlier case would make every later case look like a success.
  • Every source mutation goes through mutate, which asserts the file actually changed. Four cases were found to be dead this way: a sed that matched nothing, a helper that had been renamed, a helper that had been reformatted, and a checker that correctly stayed green because the breakage it looked for was no longer the breakage the checker hunts. Two of the four (MovAlDh, StBxDl) were repaired rather than deleted.

The one that produced the most information was restoring the original shifted mod=11 table: it turns three cells red rather than one, because the error is invisible at code 100 and only visible from 101 down. See below.

The executor problem (SOLVED — images boot and run)

The whole point of a Pascal→8086 compiler is that the output runs, and for the first four milestones it did not: no compiled image and no runtime entry had ever been executed on a CPU, correct or otherwise. Everything in the checks above was a claim about bytes. Whether the bytes work was the untested part, and it was the only part that could not be closed by writing another checker.

It is now closed, and the milestone is v-TP3-EXECUTION. 21 fixtures compile to .COM images, are booted on a floppy by a 512-byte hand-assembled boot sector, and are compared byte for byte against the exact output the fixture demands — tests/run_com_exec.py, wired into run_all.sh, 21/21 at that tag and 30/30 now.

Everything below is about establishing what can be believed, because the first attempt at this used an emulator that was wrong, and a wrong oracle is worse than none: it cannot distinguish "my codegen is broken" from "the machine is broken".

Unicorn 2.1.4 cannot be used for this. UC_MODE_16 mis-decodes 16-bit ModRM memory operands. The measurement, by loading a byte-pattern image so a load reveals its own effective address, then executing a single LEA AX,[r+disp8] and reading AX back with every register set to a distinct value:

mod=01, disp8=4            got        8086 says
  8D 40 04  LEA AX,[BX+4]  AX=0d04    0x504   (= BX+SI+4)   WRONG
  8D 41 04  LEA AX,[BX+SI+4] AX=0e04  0xd04                WRONG
  8D 42 04  LEA AX,[BX+DI+4] AX=0f04  0xe04                WRONG
  8D 43 04  LEA AX,[BP+4]  AX=1004    0x704                WRONG
  8D 45 04  LEA AX,[DI+4]  AX=0904    0x904                right
  8D 46 04  LEA AX,[BP+4]  AX=0704    0x704                right
  8D 47 04  LEA AX,[DI+4]  AX=0504    0x904                WRONG
mod=00 / mod=10 direct disp16
  8B 1E 00 20  MOV BX,[2000]  BX=1234                     right
  8D 06 34 12  LEA AX,[1234]  AX=1234                     right

So rm=5 and rm=6 decode correctly but rm=0,1,2,3,7 do not, and only base-register-free addressing (direct disp16) is trustworthy. That rules Unicorn out as an oracle for exactly the instruction forms generated code is made of — LEA AX,[BP+d], MOV AX,[SI+d], LODSW-style loops, everything with a frame pointer. It cannot distinguish "my codegen is wrong" from "the emulator is wrong", which makes it worse than no emulator at all.

pip install --upgrade unicorn resolves to the same 2.1.4, so this is not avoidable by upgrading.

This is no longer the whole picture: qemu-system-i386 is now a trusted oracle, and it was trusted by measurement, not by reputation. modrm19.s assembles with GNU as, is wrapped in a 512-byte boot sector, and is booted under /usr/bin/qemu-system-i386 with the serial port captured to a file. For each of 24 ModR/M encodings it stores a marker through the encoding under test and then scans memory for where the word landed, with BX=1000 DI=2000 SI=0030 BP=0040 so every candidate address is distinct. The answers come out as raw offsets, so this is arithmetic, not a judgement call, and the 23 cells it covers are the project's ground truth for effective addressing. It is also how we know qemu's 8086 is right where Unicorn's is wrong, on the same instruction class, by the same method.

Two facts about the tooling came out of that work and are worth recording because both were believed wrong at first:

  • objdump -D -b binary -m i8086 disassembles 16-bit code correctly. An earlier note in this file said there was no usable 16-bit disassembler available; that was wrong, and the cost of believing it was a hand-derived ModR/M table. FCML (fcml-disasm -m16) is the other decoder and the two are cross-checked by tests/fcml_vs_objdump.py. Note fcml-disasm linear-sweeps and aborts (rc=134) on inputs of 16 bytes or more through the Debian wrapper, which is why tests/disasm16.py windows input at 15.
  • A qemu execution probe for mod=11 is structurally impossible, not merely awkward. The comparison register is itself a candidate target, and SP is destroyed by the next call before any check can run — the first version of that probe pushed its return address through SS:0xBEEF, so the evidence it was about to collect had already been overwritten. It also "cleared AX because AX is never an r/m target", which is true of the wrong table and false of the right one. So mod=11 is measured by encoding instead, with as as an oracle independent of both the runtime and qemu.

Also ruled out, for the record:

  • DOSBox-X 2025.02.01 (installed, /usr/bin/dosbox-x, plain root-owned ELF, not a snap) starts cleanly headless under SDL_VIDEODRIVER=dummy SDL_AUDIODRIVER=dummy, but its -c/autoexec commands never observably execute: a md never appeared on the host, and a .COM that creates OUT.TXT via INT 21h AH=3Dh/40h never produced the file. Shell > is intercepted by dosbox-x's own wrapper (SHELL:Redirect output to out.txt). A .BAT route timed out. The user has since installed FreeDOS (freedos.qcow2, FD14-LiveCD) which is very likely the answer to this, and untried.

The boot sector — tests/exec/bootcom.s

The missing piece is now written. A 512-byte boot sector assembled with GNU as and objcopy-ed onto a 1.44 MB floppy image: it loads sector 0, reads the .COM off the same floppy with INT 13h AH=02h (disk geometry C0 H2 S18, 512-byte sectors, so .COM byte n is at disk offset 512 + n — file offset 512 maps to memory 0x100), sets DS=ES=0, SS=2000h, SP=2004h, builds the three GDT descriptors at 2000h/2002h/2004h (code, data, stack) in the way a .COM expects, installs an INT 21h shim that routes AH=02h/09h/4Ch/08h to the serial port at COM1, and JMP 0000:0100. qemu is invoked as qemu-system-i386 -fda disk.img -serial out.txt, and the fixture's expected text is compared to out.txt exactly — including CRLF, and including the exit code the program passes to INT 21h AH=4Ch.

The INT 21h shim has two distinct epilogues for one hook, which is a fact about the 8086 and not a design choice: AH=08h (read with no echo, EOF) must be able to return CF=1 with AL=0, so it exits through a different path (.Ldone8) from the AH=02h/09h/4Ch case. And the runtime's INCUR is an index into the input buffer, not a pointer, so the shim's EOF comparison is an index comparison — getting that wrong produces an "EOF at the first character" bug that looks exactly like a broken readln.

run_com_exec.py --show prints the serial file, so a failing fixture can be told apart from a broken harness without re-deriving anything.

The runtime, entry by entry — tests/rt_exec.py

run_com_exec.py proves emitted programs behave. This harness proves the runtime entries themselves do, which is a different question: a program never calls rdint without a readln behind it, never calls rdbool twice in a row, and never exercises the INT16 extremes individually.

35 cases, one fresh qemu boot each, plus a pre-flight that runs before any of them: initmem (data poisoned to AA first, so a zero is a result and not a leftover), wrint ×10 including both INT16 extremes, wrchar ×2, wrbool ×3, wrln, stackchk, a composed writeln(42) writeln TRUE sequence, rdint ×6 against supplied input, rdchar ×2, rdbool ×6, rdint at EOF, and wrtinl.

Three properties are load-bearing, and each of them was chosen against the easier alternative:

  • One boot per case. A machine that has already run a case has already run a runtime entry, and if that entry corrupts something the next case inherits it. About a tenth of a second a boot buys a machine that has provably never executed anything, and 35 of them cost under four.
  • One case record, written twice. rt_exec.py and exec/rtdrv.s each state the 40-byte layout, and the cases only pass if the two agree on every offset. A driver that silently read the wrong field would still boot, still run, and still print — so the duplication is the check.
  • A BP-contract pre-flight, before any machine starts. See below.

The harness reuses tests/exec/bootcom.s, the same 512-byte boot sector run_com_exec.py uses, so the boot machinery is written once.

Two expectations were wrong, and both were wrong in the harness's favour

Recording this because the instinct on a red test is to suspect the code, and in both cases the code was right:

  • rdchar stores one byte (StDiDl = MOV [DI],DL), so the old check dumped a two-byte word and compared it against ord(c). It could never pass. The case now reports two bytes and expects the character followed by the 0xEE poison still sitting in the second.
  • rdint at end of input stores nothing — TP3's xrdint returns to rnerr without touching the variable, and EmitRdInt's own comment says so. The old expectation said the variable was set to zero.

wrtinl also needed a harness change rather than a machine change: that entry's argument is not a stack word, the caller must place a length byte and the characters at the return address, so testing it also tests the encoding contract between Compiler.IoCall and Runtime.EmitWrInl.

What execution found that no byte check could

A 10-byte corruption of the BIOS data area. The boot sector DMA'd the image straight to 0000:0100, which is what DOS does — and 0400h-04FFh is the BDA, where SeaBIOS keeps live state. The transfer destroys it, SeaBIOS writes part of it back after the DMA, and ten bytes of BIOS data end up on top of the image. The read sets CF=0 and returns success. The image is the right length in the right place and ten bytes of it are wrong.

It was found by a 19-byte probe that read 0440h as its first instruction after the jump and got the BIOS's own bytes back, and then localised by dumping the whole loaded image against a pattern: one 10-byte run wrong inside an otherwise byte-perfect sector, which no partial-read or sector-count bug can produce. The fix is in bootcom.s: read to 8000h — clear of the IVT, the BDA, SeaBIOS's stack at 700h and the ROM window at C000h — and then REP MOVSW down to 0100h. The copy is executed code, so it is the last thing that touches the image and no BIOS call follows it.

CmpAl (20) was decimal twenty. rdint's lead-in skip is CmpAl (20H), JBE — skip everything at or below a space. Written as 20, Modula-2 read it as decimal and emitted 3C 14, so a leading space was never skipped and the scan ended with nothing read. Every other magic number in Runtime.mod is now written as hex (MovAh (02H), CmpAl (0DH), CmpAl (1AH)), because a literal that reads like hex but is decimal is silent.

wrchar and wrbool destroyed BP. Both borrow BP to reach their argument — [SP] cannot be encoded in 16-bit mode, so BP stands in — and neither saved it. BP is the one register an entry may keep, because the driver keeps its cursor into the case record there. So wrchar sent the next call to a garbage address, the machine triple-faulted, SeaBIOS rebooted, and the run printed the record header twice and hung. Symptom: record was never closed (EOT).

The interesting part is that every existing check called that shape correct. The bytes were well formed, the size did not change, the golden matched, all branch targets were on instruction boundaries, and the name audit said every helper emitted what its name said. check_runtime.py's entry golden had blessed it in five bytes: "wrchar": "8B EC 8A 46 02 89 EC". A positional golden blesses whatever is there.

So the rule is now stated, in two places that can disagree:

  • check_runtime.py's entry goldens carry the PUSH BP and the POP BP. What the bytes must be.
  • rt_exec.py has a check_bp_contract pre-flight over the built blob. What the bytes must mean. Its two halves are not equally strong and the code says so: "must start with 55 8B EC" is exact, and "must contain a 5D" is a screen, because 5D is also a displacement byte and this code does not disassemble. Requiring the POP to be contiguous with anything else is not an improvement — wrbool legitimately closes its frame after the INT 21h, and a rule insisting on 89 EC 5D fails a correct entry, which is worse because it teaches a reader to distrust the check.

And a third consequence, in the emitters themselves: reaching the argument at [BP+2] versus [BP+4] is a one-byte difference that reads as a plausible character, and the first version of the fix passed that displacement as a Modula-2 parameter — which is invisible to audit_helpers.py, because the audit reads the name. The emitters are now MovAlArg2/MovAlArg4 and CmpArg2W0/CmpArg4W0, with the displacement in the name, so the audit pins it: a helper called CmpArg4W0 that emitted the +2 form fails the run with "step 2: displacement is 2, name says 4".

Still missing: CmdRun

The R key of the shell is still unimplemented. TP3's R runs a .COM in place, without the DOS loader, from the same 64 KB of memory; the honest way to do that is a small in-process 8086 interpreter over the image, cross-validated against qemu on the same bytes.

Components

Shell — shell/Shell.mod, Term.mod, Posix.c, TextBuf.mod

Exact TP3.0 screen (Logged drive, Active directory, Work file, Main file, Edit Compile Run Save / Dir Quit compiler Options, Text: n bytes, Free: n bytes, >), command letters drawn bold. Keys L A W M E C R S D O Q; any other key redraws the menu, like TP3. W auto-.PAS with Loading/New File; S writes ^Z EOF and rotates the old file to .BAK (unlink+rename, TP3's order); D is a DOS *.* glob listing with k bytes free; O is the options submenu; Q confirms and prompts to save when the text changed.

E and C are wired. R is a stub — it prints "Interpreter pending".

Editor — shell/Editor.mod

WordStar-style full-screen editing over TextBuf. Status line Line n Col n Insert/Overwrite Indent X:FILENAME; text on rows 2-24.

Movement (^S/^D/^E/^X/^A/^F/^R/^C/^W/^Z, ^Q S/D/E/X/R/C/B/K/P, arrows, PgUp/PgDn/Home/End), editing (^V insert/overtype, ^G delete char, backspace joins lines, ^T/^Y word/line, ^Q-Y to EOL, ^N/CR break, TAB auto-indent to the word start above), block (^K B/T/H mark, word, show, ^K C/V/Y copy/move/delete, ^K R/W file read/write), search (^Q-F, ^Q-A, ^L repeat; options B/G/n/U/W, N = no-confirm; ^A any char, CR LF matches a line break), ^P literal control char, ^U/ESC aborts a prompt. ^K-D returns to the shell with the text still in memory and changed reported. Disk format matches TP3: CRLF plus trailing ^Z, load normalises, save re-expands.

Detail: TP3-EDITOR.md, TP3-EDITOR-PSEUDOCODE.md.

Compiler — shell/Compiler.mod

Single-pass Pascal → 8086, following TPSRC6 turbo / TPSRC7-10: one pass over the shared TextBuf emitting machine code into cbuf, a patch list for forward references, TP3-style error reporting (number + relative position), and code/data size accounting. Emitted image is a byte array (mode word, CS/DS, size words, CALL initmem, MOV BP,SP, generated code).

Working subset (v0.5): integer/char/boolean/byte scalars, constants with folding, globals, locals, value parameters, procedures and scalar-result functions, ARRAY[const..const] with constant indexing, control flow, GOTO/EXIT, the standard procedures WRITE, WRITELN, READ, READLN, HALT, and inline string literals as WRITE/WRITELN arguments.

Standard procedures are KBuiltin, not KProc, because they are not called generically. TP3 (TPSRC8 pwriteln/pwrloop/prdtyped) does not pass a descriptor to the runtime: it inspects each argument's class and emits a different call per type, so formatting is fixed at compile time and the runtime only ever sees a value. IoCall mirrors that — one call per argument, then a final call for the line break:

writeln(1)      MOV AX,1     ; PUSH AX ; CALL 20H ; ADD SP,2 ; CALL 40H
writeln('a')    MOV AX,'a'   ; PUSH AX ; CALL 28H ; ADD SP,2 ; CALL 40H
writeln('hi')                 ; CALL 70H ; 02 'h' 'i' ; CALL 40H
readln(x)       LEA AX,[0104]; PUSH AX ; CALL 48H ; ADD SP,2 ; CALL 60H

(Those TU_* names are the compiler's own; the offsets behind them are now assigned from Runtime.RT_Entry rather than written down, so the placeholder-versus-real distinction is gone. The third line is the inline-literal form, which differs in kind: no value is pushed and the ADD SP,2 is absent, because the length and the characters are the argument.)

READ/READLN push the address so the runtime can store (EmPushVarAddr: LEA AX,[BP+off] via EmBpDisp for locals, 8D 06 off for globals); a non-variable argument is ETypeErr (56), as in TP3.

Detail: TP3-COMPILER.md.

Runtime library — shell/Runtime.mod, shell/Runtime.def

The 8086 runtime, assembled byte by byte from Modula-2 — no external assembler and no checked-in binary, so the tree stays self-contained and the entry offsets are derived rather than guessed. Each emitter is one instruction with its ModRM byte spelled out in a comment so the encoding can be checked by hand against an 8086 table. This is the same approach Compiler.mod already takes (Ebyte/Eword/EmCall).

It mirrors the original's own mechanism: TPSRC7 copyrt copies the runtime into the front of the code buffer (SI=DI=0, REPZ MOVSB) and pc is then initialised past it (MOV pc,#$2D7C). Same shape here, and now actually done: pc := RT_Size, dc := RT_Size + 1000H, so the image is [runtime][program header][program code] and every emitted address is image-absolute. No relocation pass is needed — worth having paid for, since a linker that has to walk fixups is a linker that can get them wrong.

Current blob: 391 bytes, 14 entries, 37 branch targets, offsets read back out of the assembled bytes by tests/check_runtime.py rather than asserted by hand:

entry offset entry offset entry offset
initmem 0 wrint 36 rdint 167
progend 28 wrchar 97 rdchar 268
stackchk 35 wrbool 109 rdbool 289
halt 28 wrreal 132 rdln 335
wrln 140 wrtinl 148

The TU_* constants in Compiler.mod are assigned from Runtime.RT_Entry in Inittur, not written down, so a runtime edit that moves an entry cannot leave the compiler calling the old address.

progend and halt deliberately share one address (XOR AX,AX / MOV AH,4C / INT 21h / RET): the compiler already zeroes AX before progend and discards the HALT argument at compile time, so both leave with exit code 0.

Conventions, matching Compiler.IoCall exactly:

entry argument notes
WrInt/WrChar/WrBool/WrReal one 16-bit value on the stack caller pops
RdInt/RdChar/RdBool one address on the stack caller pops
WrLn/RdLn/StackChk nothing
InitMem AX = offset of the program header a register, not a stack word
ProgEnd/Halt nothing exits, code 0
WrInl nothing — reads its own text via POP BX see below

InitMem receives the program-header offset in AX (not on the stack), reads the data base and end out of the header, and zeroes that range, because Pascal leaves globals undefined. StackChk is a bare RET — range and stack checking aren't compiled in yet, and the call site sits mid-expression, so it must not touch a register.

Five emitter bugs were fixed here in one session, all found by the audit_helpers.py name-vs-decode pass described above, and all of them were invisible to everything else that was already in place:

emitter emitted actually was correct
MovSiBx 89 DC MOV SP,BX 89 DE
CmpSiBx 39 DC CMP SP,BX 39 DE
MovSiAx 8B C0 MOV AX,AX (a no-op) 8B F0
initmem zeroing loop MovAxDx loaded a value it then discarded XOR AX,AX
initmem header read +8 hdrMax +6 (hdrHeap)

The third is the instructive one, because it is the same error pointed the other way. 8B C0 reads as MOV AX,AX under the shifted ModR/M table this project shipped, and the fix looked like it should be the byte that table said was SI. It is not: for opcode 8B the reg field is the destination, so 8B F0 (reg=110=SI, r/m=000=AX) is MOV SI,AX, which is what the name asks for. Getting the direction backwards nearly caused a correct fix to be reverted.

Two label-name plus fixup list: rel8, rel16 and runtime-data addresses are all patched after the blob is placed, so nothing depends on a hand-computed displacement.

It has still never executed. The encodings are now audited, golden-pinned and non-vacuity-proved, which is a much stronger static claim than "it assembles" — but a static check cannot tell you the code works, only that it is what was intended. See the executor section.

The program image — shell/Linker.mod

The runtime is copied to the front of the code buffer, then the program header, then the program. The header is our own format at rtSz:

off field
+0 hdrFlag 1, "header present"
+2 hdrCS
+4 hdrDS
+6 hdrHeap = dc, the end of the data area
+8 hdrMax
+10… max-open-files, input buffer, output buffer words

InitMem gets the header offset in AX and reads +6 for the heap limit; the independent checker in run_com_tests.sh restates these offsets as its own constants and asserts initmem's SI displacements equal them, assertion by assertion, rather than asking the compiler where it thinks the header is.

CmdCompile honours the Destination option: 0 = memory, 1 = .COM, 2 = .CHN (refused). A .COM is named after its source with the extension swapped at the last dot, padded with a zero gap to max(pc, dc); its stack sits at the segment top (SS = SP = CS:FFFE), which is where DOS puts it.

Known limit: the data area starts at a fixed rtSz + 1000H (391 + 4096 = 4481), so a program whose code exceeds 4 KiB runs into its own data. Every fixture is at 4491 or 4493 bytes. Documented rather than fixed, because the original has the same fixed-offset behaviour.

String literals — writeln('hi')

writeln('toto') was the last thing standing between the front end and a hello-world: string literals had no encoding at all, so anything past one character was ENoLib. The encoding is not invented — it is the original's.

TPSRC8 pwrinlin peeks at the character after the literal: if it is , or ) the literal is a WRITE argument, not an expression, and it emits (TPSRC10 estring) the length byte and the characters into the code stream right behind the call:

CALL wrtinl   <length byte> <character>...

TPSRC4 xwrtinl is what makes that self-delimiting: POP BX takes the return address — which is the address of the length byte — and the entry ends with JMP BX, returning to just past the last character. So the literal needs no terminator, no length table, and nothing at all in the data segment. The arithmetic confirms it: t26 emits 02 68 69 inline and its data size is unchanged from a program with no strings at all.

wrtinl is 19 bytes at offset 148 (5B POP BX · 31 C9 XOR CX,CX · 8A 0F MOV CL,[BX] · 43 INC BX · B4 02 MOV AH,2 · E3 07 JCXZ to the end label · 8A 07 MOV AL,[BX] · CD 21 · 43 · E2 F9 LOOP · FF E3 JMP BX) — hand-checked once, and now also covered by the golden disassembly and the audit.

A string literal is a value in exactly one place: a WRITE/WRITELN argument. Everywhere else it is a hard error, and it is enforced in a single place — LoadAtom — because assignment, IF, WHILE, FOR, REPEAT, CASE, array subscripts and every operator all reach their operand through LoadAtom, and none of them can use a counted string where a 16-bit word is expected. ParseFactor therefore marks a literal (kind = 3) rather than rejecting it, and IoCall handles it before LoadAtom is ever reached. The alternative — letting it through and producing a machine word that happens to be a pointer — would be a silently wrong program; t25 pins the error instead.

Literal text has to survive from the scan to IoCall, since the parser does not yet know it is writing rather than computing, so it is collected into a pool as it is read: strPool[0..4095], strOff/strLen[0..255], strTop, strCnt, plus StrNew/StrPut. The pool is reset in Inittur, so it is per-compilation.

Details that are deliberate, not incidental:

  • '' is a zero-length literal, reaching the runtime's JCXZ path. It used to be the scalar 39, so writeln('') printed a quote mark.
  • 'don''t' — the doubled quote becomes one character; t24 pins len 5.
  • A literal of 256 characters or more is EConstRange (45), not truncated. The length is one byte, so 300 characters would go out behind a length of 44 and the runtime would print 44 of them and silently drop the rest. TP3 strings are at most 255 characters, so refusing is the faithful answer.
  • A string variable is ENoLib, not wrong code. IoCall knows the difference and refuses. EmPushVarAddr's local form is fixed now (see bug 4 below), so the blocker is no longer the encoding — it is that there is no string type, no length word, no assignment path and no WrStr entry.

Honest limitations

  • CmdRun — the R key — is still a stub. The compiler's output now executes (30 fixtures, exact output, exit codes), but the shell cannot run a .COM in place. The linker writes a real .COM and the boot sector runs one under qemu; nothing in the host program yet interprets 8086 code. So the user's route to seeing output is "compile, then run under qemu", not "press R". TP3's R runs in the same 64 KB with no DOS loader, which is why this is an interpreter and not a system() call.
  • Every fixture that compiles is now also run. 30 of 33 execute; the 3 that do not are t14 and t25 (ENoLib, by design) and uierror (a deliberate syntax error). The gap this replaces was nine fixtures that compiled and were never executed — t08 const, t09 if/then/else, t10 while, t11 for/to, t12 repeat/until, t13 procedure + value param, t15 label + goto, t27 five locals, t28 the 70-parameter declaration — and all nine contained no write/writeln at all. They assigned to a variable and fell off the end, so the only thing an empty .out could have asserted was "did not crash". That is why t12's loop ran exactly once for the whole life of the project with nothing to see it. The nine now print, each .out is derived by hand from the Pascal rather than from the machine, and nonvacuity.sh mutates each of the four bugs back in and requires the execution check to go red. Finding them cost four fixes; see the milestone section.
  • A call takes at most 16 arguments, and says "compiler overflow" if you exceed it. args is ARRAY [0..15] OF ERes in the three call parsers, and the guard raises ECompOvf = 99. So far (1, 2, ... , 70) is rejected with error 99, whose text in TP3 means the compiler's own table overflowed — an error about the compiler, for a program that merely has a long argument list. This is why t28_farparam can declare 70 parameters (which is what puts p63 at [BP+128] and exercises the disp16 encoding) but can only pass 16, and why its body reads p63/p70 into a variable whose value is deliberately absent from the .out: those two slots hold stack garbage, and a fixture that printed them would be testing the harness, not the compiler.
  • Parameters are separated by , and only by ,. procedure two (a : integer ; b : integer) is error 1 at the semicolon; procedure two (a : integer, b : integer) compiles. The reverse holds for variable declarations — var a, b : integer is error 1 at the comma and needs two var lines. Inconsistent, and neither spelling is wrong Pascal, so a program that compiles under one compiler may not under another.
  • A parameter may not shadow a global. DefProc calls DupTest on every parameter name, and DupTest reports error 41 for any name Search finds — including one declared at an outer level. procedure bump (x : integer) with a global x is rejected. Pascal allows the shadow and the inner one wins. HideLocals fixed siblings colliding with each other; this is the parent-vs-child direction of the same question and is untouched.
  • The branch and compare lowering is 386 code on an 8086 target. EmJcc emits 0F 8x rel16 and EmSetcc emits 0F 9x (SETcc). Neither exists on an 8086. TP3 uses JZ rel8 over a 3-byte EJMP, with excond materialising the negated boolean (TPSRC8 ~244-300). Every relational operator and every conditional branch in the compiler is affected, so this is not one call site but the whole idiom. No test can see it: qemu-i386 defaults to a post-386 CPU, so every image in run_com_exec.py runs correctly on hardware that did not exist when TP3 shipped. Catching it needs -cpu 8086 (untried) or a rewrite to the excond idiom plus a short/near branch policy — a milestone with a wide golden blast radius, deliberately not folded into this one.
  • The runtime's entries are now each called directly, and two of its invariants are stated rather than implied. 436 bytes, 14 entries, 100 emitter helpers decoded against their own names across both modules, the whole code region golden-pinned, all branch targets on instruction boundaries — and 35 cases that call the entries one at a time under qemu (rt_exec.py, 36/36). What is not covered is what a direct call cannot see: wrtinl is checked, but only from the harness's side of the calling contract. (The nine fixtures that compiled but were never executed used to sit here as an extra gap; they are now run, so every fixture that reaches the runtime at all also reaches it through generated code.)
  • The pushback slot's address is a moving target. It sits at rtSz + LoadBias + dataAt + D_PUSH, so every runtime growth moves it, and every address derived from it must be recomputed. It is computed, not hard-coded — but the two RT_SZ constants that were hard-coded and had drifted (see the execution section) are the precedent for why this one gets stated every time.
  • 30 fixtures is a small sample of Pascal. They cover var, const, if, while, for, repeat, case over scalars, procedures with value parameters, goto/label, string literals and readln. They do not cover nested procedures, recursion, var parameters, with, records, sets, files, reals, or any type wider than 2 bytes — all still ENoLib. A green execution matrix says nothing about those.
  • wrreal is a deliberate stub. It writes the literal text ?REAL? — the string lives in the runtime's own data block at D_REAL=24, which is what makes it a real 9-byte routine rather than a trap. Reals are not formatted yet, so writeln(1.5) "works" and prints nonsense. A trap would be louder; neither choice is a real answer, and this is now reachable code rather than an unreachable one, which raises the stakes on the choice.
  • Code above 4 KiB overruns the data area. The data base is fixed at rtSz + 1000H = 4432 and a .COM is padded to max(pc, dc), so the fixed 4 KiB code window is real and not advisory. The largest fixture (t32_forexit, 135 bytes of code) is nowhere near it, so nothing has hit this and nothing tests it. rtSz also moves every time the runtime grows, so the window shrinks silently — another derived value that must never be restated.
  • [SP] cannot be encoded on an 8086, and the fix is a shape change. EmMovAxSp/EmMovCxSp now emit POP reg / PUSH reg — two instructions where there used to be one, so anything that assumed a one-instruction emitter has to be revisited. The audit handles them as a documented sequence; if another one appears, the sequence machinery is where it goes, not a name hack.
  • String literals work; string variables do not. A literal in a WRITE/WRITELN argument list is emitted inline and needs no runtime support beyond wrtinl. Declaring s : string, assigning to it and printing it are all ENoLib — there is no string type, no length word, no assignment path, no WrStr entry. The chr flag on ERes is what keeps writeln('a') calling the character writer instead of the integer writer; without it the compiler emitted the integer path and printed 97 while the test still said OK.
  • Comma-separated names are not supported. var i, c : integer; is a parse error. Pre-existing, unrelated to any of the above, and still open.
  • readln of a BYTE is a latent 1-byte overflow. It calls rdint, which stores a 2-byte word, so the high byte lands on the next variable. TP3 has a separate xrdbyte for exactly this. No fixture declares a BYTE, which is the only reason this has never been observed. Write the fixture first.
  • The program-header parameter loop is unguarded against non-advancing input. program p(1;) would loop forever. The fix needs a BOOLEAN flag and must not use EXIT, which ICEs gm2 in pass 3 (see the pitfalls list). Not yet done.
  • Not implemented (all ENoLib): real, set, record, file, string variables, and any type wider than 2 bytes. with is ENoLib. case is implemented (cascade CMP/JNZ per label, per RESUME-TP3.md §3.6) but only over scalar labels — subrange labels and label lists are untested.
  • gm2 string-literal → ARRAY OF CHAR assignment copies the literal plus a NUL and leaves the tail untouched, so NUL-terminated tables are safe — this was checked, not assumed.

Three bugs only a byte-level dump could find

None of these produced a compile error, and two produced passing tests. All three were found by hex-dumping the emitted image and decoding it by hand — code sizes looked perfectly plausible throughout.

  1. rel16 off by 2 in every direct CALL/JMP. EmCall/EmJmpNear/ EmJcc computed the displacement from pc at a point where pc already pointed past the opcode and at the displacement field; x86 measures from the end of the instruction (pc + 2). ResolvePatches, on the forward-patched path, was already right — which is why forward gotos looked fine and backward ones did not.

  2. EmMovAxSp emitted 8B 04 with no SIB byte. ModRM 04 means "a SIB byte follows", so the CMP AX,imm16 of the next instruction was eaten as that SIB byte, and the intended MOV AX,[SP] became MOV AX,[BP+DI+disp]. Every case label comparison therefore vanished and case compiled to a chain of loads from garbage addresses — while the fixture reported OK. Correct encoding is 8B 44 24 00 ([SP] cannot use mod=00, that computes BP+SP).

  3. The dump tool's own offset column was wrong. It printed 4 nibbles through a helper that formats a byte, so every row was labelled 16× too large (0010 shown as 00000100). A misleading tool is worse than none — it corrupts any offset arithmetic done from its output.

Bugs found by actually running code

Executing the hand-assembled runtime — even under a broken emulator — and running the emitted images back through the harness paid for itself immediately, because a wrong encoding executes rather than failing to assemble. Sixteen so far, none of which a compiler diagnostic would ever have reported.

  1. B() silently truncated multi-byte opcodes. PROCEDURE B emits exactly one byte and masks with MOD 100H, so B (8BE4H) — a two-byte opcode passed as one literal — emitted just E4. MOV BP,SP was missing from every frame in the runtime, so BP stayed 0 and every BP-relative access read address 4. Found by decoding the hex dump; the byte is gone, not wrong.
  2. MOV BP,SP encoded as 8B E4, which is MOV SP,SP — a no-op. The ModRM byte is mod·64 + reg·8 + rm, and I mis-derived it. Compiler.mod's EmMovBpSp had it right all along (8B 0CH); only the new module was wrong. This is why the encoding is now written as three explicit B calls with the arithmetic in a comment rather than as one hex literal.
  3. InitMem read its argument from [SP] — the return address. The convention is a register (AX), unlike the per-argument I/O entries which do take a stack word. Caught because the data area was never cleared.
  4. EmPushVarAddr computed the wrong base register for locals, and truncated the displacement (pre-existing; now fixed). It emitted 8D 46 disp, which is LEA AX,[SI+disp8], but intended LEA AX,[BP+disp8] = 8D 45 disp — so read into a local had always addressed the wrong cell, silently. It also masked the offset with off MOD 100H, losing displacements above
    1. The global form 8D 06 off (LEA AX,[disp16]) was always correct. The fix is EmBpDisp, one procedure that owns the choice: disp8 iff off <= 127, else disp16, with no overflow branch, because disp16 covers the whole 16-bit range as a signed value and every real offset is either negative (locals, allocated down from 0FFFEh) or small-positive. It is now shared by EmLoadVar, EmStoreVar and EmPushVarAddr rather than written three times, and guarded by check_framedisp.py. The old truncation was accidentally correct across −32768..+127, which is the whole range where every real variable lives — so the bug was unreachable from any fixture that existed. t28 exists to make it reachable.
  5. A string literal was eating the rest of the source. Every multi-character literal reported its error at exactly Length() — one past the last character of the buffer — so the editor landed past the final . of the program. The scanner loop tested CurCh # quote, but its "closing quote detected" branch consumed two characters (the content character and the quote), so the cursor moved past the quote and the next condition test saw the character after the literal, was satisfied, and scanned on to end-of-buffer. The IF after the loop that was meant to consume the closing quote was unreachable for any string of two or more characters — which is exactly why writeln('a') always worked and writeln('hi') never did. Worse than a bad caret: it destroyed the parse, so anything after a literal was consumed as string contents and a genuine later error was misattributed to end-of-file.
  6. Every program lost 6 bytes to an uninitialised flag. The FOR over IoCall's argument list needed a "did this argument push a value" flag, and it was never set, so the first argument's CALL and its ADD SP,2 were skipped. Nothing looked wrong — a slightly smaller image looks more plausible, not less. Only expected.tsv pinning code sizes caught it.
  7. writeln('hi') emitted 02 69 00 — i then NUL. StrNew recorded the first character of a literal but did not advance strTop, so the first StrPut landed on top of the seeded character and overwrote it. t17_two_str is what pinned it down: its third emitted character was e, the second literal's character, which had been written into that slot.

The last three were each found by a different means — (5) by noticing that every error position was exactly the buffer length, (6) by expected.tsv, (7) by hex-dumping the image — and it is worth being precise about why all three were invisible to a check that only asks "does it compile": (5) still produced a plausible error number, (6) a plausible code size, and (7) a plausible character. Each is precisely the shape of bug a compile-only fixture ships.

  1. The [BP+off] displacement was truncated to a byte (see 4 above). Found by reading EmLoadVar and asking what off MOD 100H means for a negative local offset — at which point the answer is "correct by accident, and unreachable from any fixture that exists", which is the most expensive kind of wrong.

9–13. Five runtime emitters were one ModRM byte off — MovSiBx 89 DC, CmpSiBx 39 DC, MovSiAx 8B C0, initmem's zeroing loop, and initmem's header word. Tabulated with their correct encodings in the runtime section above. All five decoded cleanly, all five passed a structural check, and all five passed a golden disassembly. They were found by the one check that asks a question the bytes can answer on their own: does this decode to what this procedure is called?

  1. The ModR/M table in the runtime's own documentation was wrong, and it had been wrong since the runtime was written. It read CX DX BX SP BP SI DI BX — the correct list with AX dropped off the front and a duplicate BX invented at the end. Every code was therefore one too low except 100, which lands on SP either way, so the error was invisible at exactly the cell anyone would check first. This was a documentation bug only: the emitters that followed the wrong table emitted 89 DE/39 DE/ 8B F0, which are right. The table is now measured, not remembered — see tests/probe/README.md, which is the fuller account.

  2. DataBytes() returned dc, the absolute end of the data area, not a size. Every program over-reported by 256, and every expected.tsv row had been baselined to agree. The field is documented as "emitted data size in bytes", so 4 is right and 260 was wrong. Re-baselining the whole matrix is exactly the move that can turn a red suite green by hiding a bug, so it was done with the semantic argument above written into the file, and the 6-byte rows (the fixtures declaring one global) are the ones that carry the claim.

  3. build_tpshell.sh was a broken duplicate of the Makefile — it built Posix as if it were Modula-2, and ignored its own link's return code. It was never run, because the Makefile is what everyone runs, and a build script nobody runs is documentation. The specific lesson: when two things must agree, keep one.

Then the image actually ran, and eleven more appeared

The sixteen above were found with byte dumps, structural checks and one broken emulator. Booting the emitted .COM under qemu and comparing exact output is a different kind of instrument: it catches bugs whose every byte is locally correct and whose only defect is that the program goes somewhere else. Eleven more, and the two worst in the project are here.

  1. FOR was emitted as a post-test loop — the increment sat outside the body and the test came after it, so the body ran once before the first comparison. TP3's own emitfor (TPSRC1, DoEmit) tests before the body and increments inside it. Every for fixture printed one line too many, and only run_com_exec.py could say so: the byte count was unchanged, the control flow was well-formed, and the matrix stayed green.
  2. EXIT inside a for body jumped to the increment, not to the exit. t32_forexit therefore re-tested the condition and could re-enter the body. Worse, the handler had an EmAddSp (2) left over from when it exited a with-style scope, which unbalanced the argument-cleanup stack. The patch-list sweep also had to be rewritten from FOR i := a TO exitCnt - 1 to WHILE i < exitCnt — exitCnt is a CARDINAL, so an empty range means TO 65535, and a program with no EXIT in the loop patched 65532 slots. That is a MODULA-2 idiom trap, not a codegen bug.
  3. The procedure-skip jump was missing, so procedure bodies executed as part of the main body. The compiler emits a procedure's body between the caller's prologue and its own main body, so a jump over it is required. DeclaresProc () now answers "does this program declare any procedure?" by a save/restore-srcPos lookahead, and Compile emits EmJmpNear (0) after the prolog when the answer is TRUE, patching it with SetPatTgt (overProc, pc) after DefPart. Four fixture code sizes grew by exactly 3 bytes — the E9 plus its rel16 — and each re-baseline is documented in expected.tsv rather than waved through.

  4. And then the jump's patch slot could not be told from "no jump". EmJmpNear (target) returns a patch slot index, and slot 0 is a legitimate slot. The sentinel was overProc := 0, so IF overProc # 0 THEN SetPatTgt (...) skipped the patch for a program whose procedure-skip jump happened to be the first patch in the image. The jump kept its placeholder target 0, so it landed at image offset 0 — the entry jump — and the program looped forever, printing nothing. The fix is a separate hasProc : BOOLEAN, not a magic slot number. t31_procparam hung rather than failed, which is how it was found: a fixture that never terminates is a louder signal than one that prints the wrong thing — but only if the harness has a timeout, and only if somebody reads a timeout as information.

    The general lesson is the one this project keeps re-learning: a plausible placeholder is indistinguishable from a real value. Slot 0, 0 as "no target", 0 as "no flag set" — each was correct until the first case where the real value was 0. A separate BOOLEAN has no collision to have.

  5. IF MatchKey (tok) AND (tok = TkElse) consumed the token it was rejecting. AND is not short-circuit in Modula-2, and a lookahead built out of a matching primitive is a parser bug, not a lookahead. MatchKey advanced past the keyword, so an else that should have been left for the enclosing statement was eaten. The shape that works is PeekKw (tok) for the question and DropB (MatchKey (tok)) for the commit — and IF <stmt> may not be the last thing in a compound, which is a separate ISO rule that this hit too.

  6. A getbyte with no pushback slot lost one character per call. TP3's getbyte has a char pre-read flag: it reads the next character while looking for digits and then hands it back. The first port had no such slot, so rdint consumed the delimiter and the following statement lost its first character. The fix is a one-character pushback slot D_PUSH plus ungetch, and the slot's address moves whenever the runtime grows (rtSz + LoadBias + dataAt + D_PUSH — at 436 bytes, image 0x1B4, memory 0x2B4, one pad byte before the program header at 0x1B7). It is computed, never hard-coded: the same class of restated-constant bug as RT_SZ above.

  7. rdint did not mirror TP3's xrdint/readnum. TP3 checks for ^Z first, then skips characters <= 20h, then an optional sign, then digits, then pushes the terminator back. Ours skipped leading whitespace only and did not push the terminator back, so two readlns misbehaved. MUL r/m16 also writes DX, so the running digit is parked in DI — an encoding fact that has to be written down, or the next person "simplifies" it back.

  8. INT 21h AH=02h was given the character in DL. It takes it in AL. Four emitters (MovAlD, MovAlSi, MovAlArg, MovAlDl) were wrong together, so every character written was the high half of something else — and the count of bytes written was right, which is why a byte-counting check passed.

  9. EmMovAxSp / EmMovCxSp tried to encode [SP], which the 8086 has no ModRM form for. They now emit POP reg / PUSH reg — a two-instruction shape, so the audit treats them as a sequence rather than trying to match a mnemonic.

  10. There is no [BX] in mode 16. mod=00 rm=111 is [BX+SI], not [BX]. A store through a pointer was writing to BX+SI. Stores now go through DI or SI, and absolute data addresses use mod=00 rm=110 = ModRM 1E. Worth stating as a rule because the name [BX] appears in half the 8086 documentation ever written.

  11. EmMoveAxDx emitted 92h — which is XCHG AX,DX. Behaviour was identical either way, so nothing was broken; only the name lied, and a name that lies is how the next one becomes a silent wrong value. It is now EmXchgAxDx. The same pass audited the emitter vocabulary and renamed the ambiguous ones so the distinction survives: MovBxImm/MovBxVx (ADDRESS) vs LdBxVx (CONTENTS) vs StVxBx; MovAlBl (8A C3, register) vs LdAlBx (8A 07, memory); MovAh0 → {0xB4} vs MovAl0 → {0xB0}.

Then nine fixtures were made to print, and four more appeared

Twenty-seven bugs, and every one of them had been found by looking at bytes, decoding them, or reading them. The remaining fixtures — nine of them, the control-flow ones — had never been executed, because each one assigned to a variable and fell off the end: no write, no writeln, nothing to observe. An empty .out would have asserted only "did not crash", which is a property of the runtime, not of the operator under test. t12_repeat could have been compiling repeat…until as a single-pass loop for the entire life of the project and no check would have said.

Rewriting the nine to print, with each .out derived by hand from the Pascal and not blessed from the machine, found four more. All four shipped with a green compile matrix, a green .COM layout check, a green golden and a green emitter audit.

  1. a * b emitted ADD AX,CX. ParseAdd numbered + as 1 and ParseMul numbered * as 1 as well, and both pass the bare number to BinOpEmit, which cannot see which precedence level called it. So every multiplication dispatched to the addition: a * 2 became a + 2, and 7 * 6 printed 13. The constant-folding arm of BinOpEmit was correct, which is the only reason anything looked right — t08_const's n * n has two constants, and the one fixture that ever multiplied was multiplying two constants. Now OpAdd/OpSub/OpMul are named constants, so two levels cannot collide on a bare 1, and t08 was given a variable multiply.

  2. > and >= had their SETcc opcodes swapped. Op 4 (>) emitted 9Dh = SETGE and op 5 (>=) emitted 9Fh = SETG, so a > b meant a >= b and a >= b meant a > b. Only the equality boundary could see it: 6>5, 4>5, 5<5, 4<=5 and -1>-2 were all already correct, and 5 > 5 and 5 >= 5 were both wrong. One letter apart in the mnemonic, and the CASE arm gave no hint which comparison it answered. Every arm now carries its mnemonic beside the hex.

  3. REPEAT…UNTIL looped back while the condition was TRUE — JNZ where the body should be re-entered only when the condition is FALSE. That is while, so the body ran once, the condition was tested, and it stopped. i := 0; repeat i := i + 1 until i > 5; writeln (i) printed 1; the hand-derived answer is 6. TP3's own sequence (TPSRC8 ~244-300) calls excond, which materialises the negated boolean, then a fixed brnchop of JZ, so a single branch shape serves IF, WHILE and REPEAT alike.

  4. Sibling procedures shared one parameter namespace. A finished procedure's symbols were left at a level that Search's level <= lexnest test still accepted, and every procedure body compiles at the same depth. So a second a : integer was a duplicate (error 41) and an unqualified a inside procedure two silently read procedure one's argument — passing 3 into one and then computing x := a + 1 in two printed the wrong number with no diagnostic at all. HideLocals relabels a procedure's own symbols to 0FFFFH when it closes, which fails the visibility test at every depth a later procedure can be at. Relabelled rather than popped, because symtab[old].resvar holds an index and a function's result variable is one of the entries being hidden.

The theme is worth stating because it is the same theme as bug 27: a name that does not distinguish two things makes the next mistake invisible. * and + were both 1; > and >= were two hex bytes; two procedures' a was one symbol. In each case the fix is to make the distinction part of the name or the surrounding text, not to fix the value and leave the ambiguity in place.

The bug family, stated once

Nine of the thirty-one are the same bug in different clothes: loading the address where the value was wanted, or picking the register one byte or one letter away from the right one. EmPushVarAddr had the right bytes for the wrong register. LdAlBx and MovAlBl are one letter apart. MovAh0 and MovAl0 are one letter apart and their opcodes differ by 04h. The response to that is not more checking — it is that the names now carry the distinction, so the next one is a compile error instead of a silent wrong value. Check emitted bytes against intended semantics whenever an emitter is added, and let the audit do it for one-liners.

And a second family, on the harness side rather than the compiler's: a check that quietly checks nothing looks exactly like a check that passes. The audit that swept one module, the regex that matched no emitters, the coverage list derived from the parser's own output, the four nonvacuity mutations that matched nothing, and two hard-coded RT_SZ constants — six instances, all green, all worthless. That is why this file has a non-vacuity section and an independently-scanned inventory at all.

gm2 / ISO Modula-2 pitfalls hit along the way

  • Two-phase link (above) — a single whole-program pass 3 caps identifiers/errors silently.
  • EXIT inside the program-header parameter WHILE ICEs gm2 in pass 3 (ExitStatement → PopExit → M2StackWord_PopWord → invalidloc). Extracting the loop into its own procedure did not help. Use a BOOLEAN "advanced" flag, never EXIT. (LOOP+EXIT is fine, and is used widely.)
  • A bare HALT aborts under -fiso (SIGABRT, exit 134); HALT (0) is correct.
  • CHAR is not the ZType: ch = 09H must be ORD (ch) = 09H.
  • gm2's ISO SYSTEM exports ORD but not Ord — the import is case-sensitive here despite gm2's usual case-insensitivity, so FROM SYSTEM IMPORT Ord fails with "unknown symbol" while plain ORD (c) compiles. Write ORD, never Ord.
  • ISO forbids dropping a function result in a statement → the DropCh/ DropB/DropC discard helpers wrap ~39 call sites.
  • AND/OR/NOT on 16-bit CARDINAL → BitAnd/BitOr/BitNot.
  • PROCEDURE f : T is invalid; the file's convention is PROCEDURE f () : T.
  • ISO will not index a plain string constant as an array (needs an array constructor) — compute hex digits with CHR instead.
  • Foreign modules (FOR "C") must use ADDRESS, not Modula-2 pointer types; cast at the call site. Posix has no Posix.mod (do not try to rebuild it) and no argv binding.
  • termios raw mode: every newline is an explicit CR LF.
  • Foreign modules and Posix cannot pass argv; the test harness reads fixture paths from stdin instead.
  • subprocess.run(input=…) needs bytes, not str, or it raises inside Python rather than reporting the real error.
  • as always picks opcode 89 for a register-to-register mov, so it will never emit 8B EC for the mnemonic mov bp,sp. Test anchors that must assert a specific opcode have to be written as literal .byte.
  • FCML prints immediates with an h suffix (add sp,8h), and the Debian fcml-disasm wrapper aborts with rc=134 on inputs of 16 bytes or more.

Reference material

  • TP3-COMPILER.md — the compiler: what was changed, the Skip bug class, the rel16 off-by-2, standard procedures, current matrix.
  • TP3-EDITOR.md / TP3-EDITOR-PSEUDOCODE.md — the original TP3 editor, from TPSRC5/TPSRC6.
  • RESUME-TP3.md — book-derived reference for the 8086 code TP3 generates per Pascal construct (types, skeletons, arithmetic, IF/CASE/REPEAT/WHILE/ FOR, procedures, parameters, functions, I/O, typed constants, absolutes), plus the TU_* runtime entry list.
  • shell/tests/probe/README.md — read this before touching any emitter. What each ModR/M artifact establishes, which oracle measures which half of the table, why the mod=11 execution probe is structurally impossible, and the shifted table this project shipped, in full.
  • shell/tests/run_all.sh / nonvacuity.sh headers — what runs, in what order, and which breakage is supposed to turn which check red.
  • Resources/turbopascal3source/TP3/ — TPSRC1-10, the disassembled original. This is the ground truth; when our behaviour and the book disagree, the disassembly wins.
  • Resources/coeur-tp-ocr/ — OCR of "Au coeur de Turbo Pascal".

Next steps

  1. 8086 branch and compare lowering. EmJcc emits 0F 8x rel16 and EmSetcc emits 0F 9x; neither instruction exists on an 8086. Every relational operator and every conditional branch depends on them. TP3's idiom is excond materialising the negated boolean plus JZ rel8 over a 3-byte EJMP (TPSRC8 ~244-300), so this needs a boolean-materialisation strategy and a short/near branch policy, not two opcode substitutions — and it will move nearly every byte in the golden. First, try qemu-system-i386 -cpu 8086 on the existing 30 images: if that turns this class red, it is a one-line addition to rt_exec.py and it makes every later fix provable instead of argued.
  2. CmdRun as an in-process 8086 interpreter — the R menu key, and a fallback executor for environments with no DOS. Validate it against qemu on the same images, so the two oracles check each other. Cross-validation is the point: an interpreter that agrees with qemu on 30 fixtures is far more evidence than either alone.
  3. String *variables* — s : string, s := 'hi', writeln(s). The encoding blocker is gone (EmBpDisp); what is left is a length word, an assignment path, and a WrStr entry (TPSRC4 xwrtstr). IoCall currently refuses with ENoLib.
  4. Nested procedures / recursion, var parameters (the SEG:OFF push from RESUME-TP3.md §3.11), range/index checks (TU_RANGE_CHECK, TU_INDEX_CHECK), typed constants (RESUME-TP3.md §3.14), array at its point of use (t14), case with subrange labels.
  5. readln of a BYTE calls rdint, which stores 2 bytes and overflows into the next variable. TP3 has a separate xrdbyte; a TU_RdByte entry is the fix. No fixture exists yet, which is why it has not been done — write the fixture first, so the bug is red before the fix.
  6. Make the 4 KiB code window an enforced limit rather than a documented one: report an error when pc reaches dc, instead of writing over the data.
  7. Both spellings of a multi-name declaration. var i, c : integer; is error 1 at the comma and needs two var lines; procedure f (a : integer; b : integer) is error 1 at the semicolon and needs a comma. Neither is wrong Pascal, so a program that compiles under one compiler may not under another. A parameter may also not shadow a global (DupTest rejects any name Search finds at any level), which Pascal allows.
  8. Harden the program-header parameter loop against non-advancing input (program p(1;)) with a BOOLEAN flag — not EXIT, which ICEs gm2.
  9. FreeDOS (freedos.qcow2, FD14-LiveCD) is still untried. Not needed for any claim above, but it is the only way to get a real DOS as a third opinion on the INT 21h shim.