|
|
@@ -16,11 +16,12 @@ manual, not guessed.
|
|
|
| Compiler skeleton + build recipe | `v-TP3-SHELL-COMPILES` | done |
|
|
|
| Parser `Skip` bug class (9 sites) | `v-TP3-PARSER-FIXES` | done |
|
|
|
| Standard procedures + `rel16` fix | `v-TP3-STDPROCS` | done |
|
|
|
-| Runtime library + 8086 execution harness | `v-TP3-RUNTIME-BLOB` | assembled, **never run** |
|
|
|
-| Inline string literals (`writeln('hi')`) | `v-TP3-STRLITERAL` | done, **never run** |
|
|
|
-| Linker: real DOS `.COM` writer + independent byte checker | `v-TP3-COM-IMAGE` | done, **never run** |
|
|
|
-| Measured encodings: ModR/M table, runtime audit, golden disassembly, `[BP+off]` | `v-TP3-MEASURED-EMITTERS` | done, **never run** |
|
|
|
-| `CmdRun`, and a `.COM` that has actually executed | — | **not started** |
|
|
|
+| Runtime library + 8086 execution harness | `v-TP3-RUNTIME-BLOB` | assembled, executed under qemu |
|
|
|
+| Inline string literals (`writeln('hi')`) | `v-TP3-STRLITERAL` | done, executed |
|
|
|
+| Linker: real DOS `.COM` writer + independent byte checker | `v-TP3-COM-IMAGE` | done, executed |
|
|
|
+| Measured encodings: ModR/M table, runtime audit, golden disassembly, `[BP+off]` | `v-TP3-MEASURED-EMITTERS` | done, executed |
|
|
|
+| Execution: a boot sector, qemu, and a claim about behaviour | `v-TP3-EXECUTION` | done, **21/21 fixtures run, exact output** |
|
|
|
+| `CmdRun` (the `R` key), in-process 8086 interpreter | — | **not started** |
|
|
|
|
|
|
## Build
|
|
|
|
|
|
@@ -39,7 +40,7 @@ gm2 -fiso -fuse-module-list=modules.lst -o tpshell \
|
|
|
Shell.mod Compiler.mod Term.o TextBuf.o Posix.o Editor.o # phase B2
|
|
|
```
|
|
|
|
|
|
-Current clean build: `make clean && make` → rc=0, `tpshell` **163032 bytes**.
|
|
|
+Current clean build: `make clean && make` → rc=0, `tpshell` **163616 bytes**.
|
|
|
The one diagnostic is `./Compiler.mod: ParseExpr: too many errors in pass 3`,
|
|
|
which is the expected phase-1 rollup that the recipe tolerates — not a real
|
|
|
error. `shell/Makefile` is the single authoritative build recipe.
|
|
|
@@ -79,11 +80,11 @@ error position or a program that lost six bytes now fails the suite instead of
|
|
|
needing a squint. It was checked for vacuousness by reverting the string
|
|
|
scanner fix: 19/23 and exit 1, restored: 23/23 and exit 0.
|
|
|
|
|
|
-**27 of 29 fixtures compile**, up from 1 (the empty program) when the direct
|
|
|
-harness was first built.
|
|
|
+**30 of 33 fixtures compile clean**, up from 1 (the empty program) when the
|
|
|
+direct harness was first built.
|
|
|
|
|
|
```
|
|
|
-compile matrix: 29 passed, 0 failed (of 29)
|
|
|
+compile matrix: 33 passed, 0 failed (of 33)
|
|
|
```
|
|
|
|
|
|
Compiling: `t01` minimal · `t04` var+assign+`writeln` · `t06` two args ·
|
|
|
@@ -93,16 +94,20 @@ Compiling: `t01` minimal · `t04` var+assign+`writeln` · `t06` two args ·
|
|
|
`t19` `writeln(1)` · `t20` 3 string args · `t21` mixed args ·
|
|
|
`t22` `case` with two labels · `t23` `writeln('')` · `t24` `writeln('don''t')` ·
|
|
|
`t26` mixed scalar/string args · `t02`/`t03`/`t05`/`t17` multi-char literals ·
|
|
|
-`t27` five locals · `t28` a 70-parameter declaration.
|
|
|
-
|
|
|
-`comtest` additionally links **every one of the 29** to a real `.COM` and
|
|
|
-re-verifies the bytes with an independent checker that restates the layout
|
|
|
-constants instead of asking the compiler: `26 checked, 0 failed`.
|
|
|
+`t27` five locals · `t28` a 70-parameter declaration · `t29` `readln` from a
|
|
|
+supplied input file · `t30` a counted `for` whose bounds come from an
|
|
|
+expression · `t31` a value parameter *and* a call across statements ·
|
|
|
+`t32` `EXIT` out of a `for` body.
|
|
|
|
|
|
Failing, all deliberately: `t14` `array [1..5] of integer` at its point of use
|
|
|
and `t25` a string literal used as a *value* (`s := 'hi'`) both → `ENoLib`
|
|
|
(102), the original's "not implemented" path; `uierror` is a deliberate syntax
|
|
|
-error used by the UI test.
|
|
|
+error (41) used by the UI test.
|
|
|
+
|
|
|
+`run_com_tests.sh` additionally links **all 30 that compile** to a real `.COM`
|
|
|
+and re-verifies the bytes with an independent Python checker that *measures*
|
|
|
+the layout instead of restating it: `30 checked, 0 failed`. That last part was
|
|
|
+itself a bug fix — see "the two restated constants" below.
|
|
|
|
|
|
### Shell + editor UI — `shell/tests/uitest.py`
|
|
|
|
|
|
@@ -125,7 +130,7 @@ changing `Run`'s signature, so `Editor.def` stays additive.
|
|
|
feature: the latter move as the compiler grows, and a test whose
|
|
|
expectations drift with it stops being a test.
|
|
|
|
|
|
-### The encodings — five checks that can each go red
|
|
|
+### The encodings — six checks that can each go red
|
|
|
|
|
|
Nothing above looks at *machine code*. It all stops at "the compiler produced
|
|
|
what it intended to produce", which is exactly where the bugs in this project
|
|
|
@@ -139,9 +144,10 @@ proves each one can go red.
|
|
|
|---|---|---|
|
|
|
| `probe/run_modrm19.py` | the mod=00/01/10 effective addresses, by **executing** 23 cases on a real 8086 under qemu and scanning for where the marker landed | mod=11 — see below |
|
|
|
| `probe/modrm11.py` | the mod=11 register identities, by **encoding** with GNU `as` and decoding with FCML, against hard-coded bytes | the table agreeing with itself |
|
|
|
-| `audit_helpers.py` | every one-line emitter in `Runtime.mod` decodes to what its *name* says — 61/61 | anything longer than one instruction |
|
|
|
-| `check_runtime.py` + `runtime.golden` | the built runtime's 360-byte code region sweeps cleanly through FCML, every entry and all 37 branch targets land on an instruction boundary, and the whole disassembly is byte-for-byte the committed golden | whether the golden is *right* |
|
|
|
+| `audit_helpers.py` | every one-line emitter in **both** `Runtime.mod` and `Compiler.mod` decodes to what its *name* says — **98/98**, from an inventory scanned independently of the parser | anything longer than one instruction |
|
|
|
+| `check_runtime.py` + `runtime.golden` | the built runtime's code region (432 bytes, code ends at 401) sweeps cleanly through FCML, every entry and all branch targets land on an instruction boundary, and the whole disassembly is byte-for-byte the committed golden | whether the golden is *right* |
|
|
|
| `check_framedisp.py` | `[BP+off]` uses disp8 iff `off <= 127`, for locals (negative) and far parameters (>127) | which of the two encodings was chosen, if the other also works |
|
|
|
+| `run_com_exec.py` | the **emitted image executes** and prints exactly the expected bytes | semantics the fixture never exercises |
|
|
|
|
|
|
Two of these deserve the detail, because the reason they exist is the reason
|
|
|
they are hard.
|
|
|
@@ -155,6 +161,27 @@ Only asking "does this byte sequence mean what this procedure is called?" fails,
|
|
|
which is what the audit does: it disassembles each one-line emitter and compares
|
|
|
the decode against the name. Five bugs came out of it in one pass.
|
|
|
|
|
|
+And then the audit itself turned out to have **three independent ways of
|
|
|
+silently dropping subjects**, which is worse, because a check that quietly
|
|
|
+checks nothing looks exactly like a check that passes:
|
|
|
+
|
|
|
+1. It swept `Runtime.mod` only — while `EmXchgAxCx` lives in `Compiler.mod` and
|
|
|
+ was wrong (`93h` = `XCHG AX,BX`) for its entire life, with a correct byte
|
|
|
+ count and a green matrix. The fix sweeps both modules.
|
|
|
+2. The procedure-header regex required a *non-empty* parameter list, so it
|
|
|
+ matched **0 of `Compiler.mod`'s 22 emitters**.
|
|
|
+3. The coverage list was *derived from the parser's own output*. That is
|
|
|
+ vacuous: any input that stops the parser also deletes the helper from the
|
|
|
+ inventory, so the audit reports "100% covered" for a file it read nothing
|
|
|
+ from. The inventory is now a deliberately dumb `^PROCEDURE\s+(\w+)` scan
|
|
|
+ with no parameter, `BEGIN` or comment awareness, and the audit fails if a
|
|
|
+ documented emitter is missing from it. `EmitData` is the one named
|
|
|
+ exclusion, with the reason recorded in the source.
|
|
|
+
|
|
|
+`audit_vx_helpers` had a smaller version of the same disease: it *printed*
|
|
|
+`len(VX_KINDS)` = "4 Vx subjects" for `Compiler.mod`, which has none. It now
|
|
|
+reports the number actually checked.
|
|
|
+
|
|
|
**`modrm11.py` is deliberately not self-referential.** The obvious way to check
|
|
|
a ModR/M table is to write the table in assembly and assemble it — but that can
|
|
|
never fail, because editing the assembly makes `as` faithfully re-encode the new
|
|
|
@@ -177,29 +204,82 @@ entire 16-bit range as a signed value. The check also asserts the *rule* rather
|
|
|
than one encoding — always-disp16 is accepted, and `nonvacuity.sh` proves that
|
|
|
by building it and requiring the check to stay green.
|
|
|
|
|
|
+### Execution under qemu — `tests/run_com_exec.py`
|
|
|
+
|
|
|
+The sixth check is the one that cannot be written as a byte comparison, so it
|
|
|
+is also the one that finds the most: 21 fixtures are compiled to `.COM`, put on
|
|
|
+a floppy, booted, and their serial output compared to a committed `.out` file
|
|
|
+**exactly** — CRLF included — plus the exit code passed to `INT 21h AH=4Ch`.
|
|
|
+
|
|
|
+```
|
|
|
+execution: 21 passed, 0 failed (of 21)
|
|
|
+```
|
|
|
+
|
|
|
+The two fixtures it found nothing in are the interesting ones: `t31_procparam`
|
|
|
+and `t32_forexit` were the two most expensive bugs in the project, and neither
|
|
|
+was visible as a wrong byte count. See `overProc` below.
|
|
|
+
|
|
|
+**The `.COM` layout constants are measured, not restated.** `run_com_tests.sh`
|
|
|
+and `comtest.py` both used to hard-code `RT_SZ = 391` against a runtime that
|
|
|
+had since grown to 432 bytes, so they read the program header 41 bytes early
|
|
|
+and reported **30 false failures** — a red suite that meant nothing, which is
|
|
|
+the most expensive kind of red. Both now locate the header by its own
|
|
|
+signature (`hdrFlag = 1`, `hdrDS == hdrOff + 1000h + bias`, `hdrHeap > hdrDS`,
|
|
|
+`hdrCS` leaves room, `initmem` length at `ENT_SZ`) and *derive* `rtSz`,
|
|
|
+`prologAt` and `dataBase` per file:
|
|
|
+
|
|
|
+```
|
|
|
+measured runtime size: 432 bytes (header at image offset 435)
|
|
|
+```
|
|
|
+
|
|
|
+A restated constant that has drifted is worse than a derived one, and the two
|
|
|
+checkers had drifted from each other as well as from the runtime — which is why
|
|
|
+there are two of them and why both were wrong in the same way.
|
|
|
+
|
|
|
### Non-vacuity — `tests/nonvacuity.sh`
|
|
|
|
|
|
-Every assertion above is proved able to fail: **16 deliberate breakages, each
|
|
|
-asserted to turn exactly one named check red for the stated reason**, then
|
|
|
-restored and re-asserted green. Six break the runtime, five attack the mod=11
|
|
|
-table (including restoring the exact wrong table this project once shipped),
|
|
|
-and three target `EmBpDisp` — the truncation, the always-disp16
|
|
|
-over-encoding that must *stay* green, and the restored source.
|
|
|
+Every assertion in this file is proved able to fail: **28 deliberate
|
|
|
+breakages, each asserted to turn exactly one named check red for the stated
|
|
|
+reason, then restored and re-asserted green.** Six break the runtime, five
|
|
|
+attack the mod=11 table (including restoring the exact wrong table this
|
|
|
+project once shipped), three target `EmBpDisp` — the truncation, the
|
|
|
+always-disp16 over-encoding that must *stay* green, and the restored source —
|
|
|
+six attack the helper audit, and five attack the `.COM` layout checker.
|
|
|
+
|
|
|
+Two properties of the harness itself are enforced, because both had already
|
|
|
+gone wrong silently:
|
|
|
+
|
|
|
+- **A baseline assertion runs first**, so a case that is *already* red is
|
|
|
+ reported as `NOT NON-VACUOUS` and distinguished from one that *went* red.
|
|
|
+ Without it, a harness broken by an earlier case would make every later case
|
|
|
+ look like a success.
|
|
|
+- **Every source mutation goes through `mutate`, which asserts the file
|
|
|
+ actually changed.** Four cases were found to be dead this way: a `sed`
|
|
|
+ that matched nothing, a helper that had been renamed, a helper that had
|
|
|
+ been reformatted, and a checker that correctly stayed green because the
|
|
|
+ breakage it looked for was no longer the breakage the checker hunts. Two
|
|
|
+ of the four (`MovAlDh`, `StBxDl`) were repaired rather than deleted.
|
|
|
|
|
|
The one that produced the most information was restoring the original shifted
|
|
|
-table: it turns **three** cells red rather than one, because the error is
|
|
|
-invisible at code 100 and only visible from 101 down. See below.
|
|
|
+mod=11 table: it turns **three** cells red rather than one, because the error
|
|
|
+is invisible at code 100 and only visible from 101 down. See below.
|
|
|
+
|
|
|
+## The executor problem (SOLVED — images boot and run)
|
|
|
|
|
|
-## The executor problem (blocking everything downstream)
|
|
|
+The whole point of a Pascal→8086 compiler is that the output *runs*, and for
|
|
|
+the first four milestones it did not: **no compiled image and no runtime entry
|
|
|
+had ever been executed on a CPU**, correct or otherwise. Everything in the
|
|
|
+checks above was a claim about bytes. Whether the bytes work was the untested
|
|
|
+part, and it was the only part that could not be closed by writing another
|
|
|
+checker.
|
|
|
|
|
|
-The whole point of a Pascal→8086 compiler is that the output *runs*, and it
|
|
|
-still has not: **no compiled image and no runtime entry has ever been executed
|
|
|
-on a CPU**, correct or otherwise. Everything in the five checks above is a claim
|
|
|
-about bytes. Whether the bytes work is the untested part, and it is the only
|
|
|
-part that cannot be closed by writing another checker.
|
|
|
+**It is now closed, and the milestone is `v-TP3-EXECUTION`.** 21 fixtures
|
|
|
+compile to `.COM` images, are booted on a floppy by a 512-byte hand-assembled
|
|
|
+boot sector, and are compared **byte for byte** against the exact output the
|
|
|
+fixture demands — `tests/run_com_exec.py`, wired into `run_all.sh`, 21/21.
|
|
|
|
|
|
-The rest of this section is about establishing what *can* be believed, because
|
|
|
-the first attempt at this used an emulator that was wrong, and a wrong oracle is
|
|
|
+Everything below is about establishing what *can* be believed, because the
|
|
|
+first attempt at this used an emulator that was wrong, and a wrong oracle is
|
|
|
worse than none: it cannot distinguish "my codegen is broken" from "the machine
|
|
|
is broken".
|
|
|
|
|
|
@@ -276,24 +356,53 @@ Also ruled out, for the record:
|
|
|
since installed **FreeDOS** (`freedos.qcow2`, FD14-LiveCD) which is very
|
|
|
likely the answer to this, and untried.
|
|
|
|
|
|
-**So what is still missing is narrow: qemu can decode, assemble and execute, but
|
|
|
-nothing has yet booted an image that was produced by *this* compiler.** The
|
|
|
-remaining piece is a boot sector that reads a `.COM` off the floppy with
|
|
|
-`INT 13h`, sets `SS:SP` at the segment top, hooks `INT 21h` for
|
|
|
-`AH=02h/09h/4Ch` to the serial port, and `JMP 0x100`. `tests/rt_exec.py` is
|
|
|
-the harness that will consume it: it loads the runtime, calls each entry with a
|
|
|
-known argument, and compares the bytes sent to `INT 21h` against expectations —
|
|
|
-33 checks covering `initmem`, `wrint` (10 values incl. both `INT16` extremes),
|
|
|
-`wrchar`, `wrbool`, `wrln`, `stackchk`, a composed `writeln(42) writeln TRUE`
|
|
|
-sequence, and the read entries against supplied input including EOF. It was
|
|
|
-written against Unicorn and **currently fails 33 of 33** — the failures are
|
|
|
-Unicorn's, not the library's, and `run_all.sh` does not run it. Re-point it at
|
|
|
-qemu and those numbers become a verdict on the runtime. It has **no `wrtinl`
|
|
|
-case yet**, which needs a harness change rather than just a machine change: that
|
|
|
-entry's argument is not a stack word, the caller must place a length byte and
|
|
|
-the characters at the *return address*, so testing it also tests the encoding
|
|
|
+### The boot sector — `tests/exec/bootcom.s`
|
|
|
+
|
|
|
+The missing piece is now written. A **512-byte boot sector** assembled with
|
|
|
+GNU `as` and `objcopy`-ed onto a 1.44 MB floppy image: it loads sector 0,
|
|
|
+reads the `.COM` off the same floppy with `INT 13h AH=02h` (disk geometry
|
|
|
+`C0 H2 S18`, 512-byte sectors, so `.COM` byte *n* is at disk offset
|
|
|
+`512 + n` — **file offset 512 maps to memory `0x100`**), sets `DS=ES=0`,
|
|
|
+`SS=2000h`, `SP=2004h`, builds the three GDT descriptors at
|
|
|
+`2000h/2002h/2004h` (code, data, stack) in the way a `.COM` expects,
|
|
|
+installs an `INT 21h` shim that routes `AH=02h/09h/4Ch/08h` to the
|
|
|
+serial port at `COM1`, and `JMP 0000:0100`. qemu is invoked as
|
|
|
+`qemu-system-i386 -fda disk.img -serial out.txt`, and the fixture's expected
|
|
|
+text is compared to `out.txt` exactly — including CRLF, and including the
|
|
|
+exit code the program passes to `INT 21h AH=4Ch`.
|
|
|
+
|
|
|
+The `INT 21h` shim has **two distinct epilogues for one hook**, which is
|
|
|
+a fact about the 8086 and not a design choice: `AH=08h` (read with no echo,
|
|
|
+EOF) must be able to return `CF=1` with `AL=0`, so it exits through a
|
|
|
+different path (`.Ldone8`) from the `AH=02h/09h/4Ch` case. And the runtime's
|
|
|
+`INCUR` is an **index into the input buffer**, not a pointer, so the shim's
|
|
|
+EOF comparison is an index comparison — getting that wrong produces an
|
|
|
+"EOF at the first character" bug that looks exactly like a broken `readln`.
|
|
|
+
|
|
|
+`run_com_exec.py --show` prints the serial file, so a failing fixture can be
|
|
|
+told apart from a broken harness without re-deriving anything.
|
|
|
+
|
|
|
+### Still missing: `CmdRun`
|
|
|
+
|
|
|
+`tests/rt_exec.py` is the *other* harness — 33 direct calls into the runtime
|
|
|
+entries (`initmem`, `wrint` ×10 including both `INT16` extremes, `wrchar`,
|
|
|
+`wrbool`, `wrln`, `stackchk`, a composed `writeln(42) writeln TRUE`
|
|
|
+sequence, and the read entries against supplied input including EOF). It was
|
|
|
+written against Unicorn and **fails 33 of 33**; the failures are Unicorn's,
|
|
|
+not the library's, so `run_all.sh` does not run it. Re-pointing it at
|
|
|
+`qemu-system-i386` — reusing the very same `tests/exec/bootcom.s`, so the
|
|
|
+boot machinery is written once — turns those numbers into a verdict on the
|
|
|
+runtime rather than on the emulator. It has **no `wrtinl` case yet**, which
|
|
|
+needs a harness change rather than just a machine change: that entry's
|
|
|
+argument is not a stack word, the caller must place a length byte and the
|
|
|
+characters at the *return address*, so testing it also tests the encoding
|
|
|
contract between `Compiler.IoCall` and `Runtime.EmitWrInl`.
|
|
|
|
|
|
+And the `R` key of the shell is still unimplemented. TP3's `R` runs a `.COM`
|
|
|
+*in place*, without the DOS loader, from the same 64 KB of memory; the
|
|
|
+honest way to do that is a small in-process 8086 interpreter over the
|
|
|
+image, cross-validated against qemu on the same bytes.
|
|
|
+
|
|
|
## Components
|
|
|
|
|
|
### Shell — `shell/Shell.mod`, `Term.mod`, `Posix.c`, `TextBuf.mod`
|
|
|
@@ -542,19 +651,44 @@ Details that are deliberate, not incidental:
|
|
|
|
|
|
## Honest limitations
|
|
|
|
|
|
-- **A compiled image has still never been executed.** This is the one
|
|
|
- limitation that everything else is downstream of. `CmdRun` is a stub; the
|
|
|
- linker *does* now write a real `.COM`; qemu is a proven oracle for encodings
|
|
|
- but nothing has yet booted an image this compiler produced. Everything in the
|
|
|
- checks section is a claim about the bytes, and the bytes have been checked
|
|
|
- hard. Whether the bytes *work* is exactly the untested part.
|
|
|
-- **The runtime is audited but unproven.** 391 bytes, 14 entries, every one-line
|
|
|
- emitter decoded against its own name, the whole code region golden-pinned, all
|
|
|
- 37 branch targets on instruction boundaries — and still never run on a CPU.
|
|
|
- Nine of its own bugs have been found this way so far, so the prior is not
|
|
|
- reassuring. `wrtinl` is the newest entry and the only one no check touches
|
|
|
- even in principle: its argument lives at its own return address, so testing
|
|
|
- it needs a harness that models the caller's contract.
|
|
|
+- **`CmdRun` — the `R` key — is still a stub.** The compiler's *output* now
|
|
|
+ executes (21 fixtures, exact output, exit codes), but the shell cannot run a
|
|
|
+ `.COM` in place. The linker writes a real `.COM` and the boot sector runs one
|
|
|
+ under qemu; nothing in the host program yet interprets 8086 code. So the
|
|
|
+ user's route to seeing output is "compile, then run under qemu", not "press
|
|
|
+ `R`". TP3's `R` runs in the same 64 KB with no DOS loader, which is why this
|
|
|
+ is an interpreter and not a `system()` call.
|
|
|
+- **21 of 33 fixtures execute.** 3 do not compile (`t14` and `t25` by design
|
|
|
+ as `ENoLib`, `uierror` deliberately), leaving **9 that compile and are never
|
|
|
+ run**: `t08` const · `t09` if/then/else · `t10` while · `t11` for/to ·
|
|
|
+ `t12` repeat/until · `t13` procedure + value param · `t15` label + goto ·
|
|
|
+ `t27` five locals · `t28` the 70-parameter declaration. Those are not
|
|
|
+ incidental omissions — they are the *control-flow* fixtures, and `t27`'s
|
|
|
+ five locals are precisely the `[BP+off]` paths this project got wrong twice.
|
|
|
+ They have byte-level checks and no behavioural check at all. Writing nine
|
|
|
+ `.out` files is cheap and is the highest-value step after `rt_exec.py`; the
|
|
|
+ byte counts in `expected.tsv` will then have a behavioural counterpart.
|
|
|
+- **The runtime is audited, executed end to end, but not per entry.** 432
|
|
|
+ bytes, 14 entries, 98 emitter helpers decoded against their own names across
|
|
|
+ both modules, the whole code region golden-pinned, all branch targets on
|
|
|
+ instruction boundaries — and reached only through the 21 fixtures' call
|
|
|
+ sequences. `rt_exec.py` exists to call all 14 directly and currently fails
|
|
|
+ 33/33 *because its machine is wrong*, so there is currently no per-entry
|
|
|
+ verdict. `wrtinl` is the newest entry and the only one no check touches even
|
|
|
+ in principle: its argument lives at its own return address, so testing it
|
|
|
+ needs a harness that models the caller's contract.
|
|
|
+- **The pushback slot's address is a moving target.** It sits at
|
|
|
+ `rtSz + LoadBias + dataAt + D_PUSH`, so every runtime growth moves it, and
|
|
|
+ every address derived from it must be recomputed. It is computed, not
|
|
|
+ hard-coded — but the two `RT_SZ` constants that *were* hard-coded and had
|
|
|
+ drifted (see the execution section) are the precedent for why this one gets
|
|
|
+ stated every time.
|
|
|
+- **21 fixtures is a small sample of Pascal.** They cover `var`, `const`,
|
|
|
+ `if`, `while`, `for`, `repeat`, `case` over scalars, procedures with value
|
|
|
+ parameters, `goto`/`label`, string literals and `readln`. They do **not**
|
|
|
+ cover nested procedures, recursion, `var` parameters, `with`, records, sets,
|
|
|
+ files, reals, or any type wider than 2 bytes — all still `ENoLib`. A green
|
|
|
+ execution matrix says nothing about those.
|
|
|
- **`wrreal` is a deliberate stub.** It writes the literal text `?REAL?` — the
|
|
|
string lives in the runtime's own data block at `D_REAL=24`, which is what
|
|
|
makes it a real 9-byte routine rather than a trap. Reals are not formatted
|
|
|
@@ -562,12 +696,17 @@ Details that are deliberate, not incidental:
|
|
|
neither choice is a real answer, and this is now reachable code rather than
|
|
|
an unreachable one, which raises the stakes on the choice.
|
|
|
- **Code above 4 KiB overruns the data area.** The data base is fixed at
|
|
|
- `rtSz + 1000H` = 4481 and a `.COM` is padded to `max(pc, dc)`, so the fixed
|
|
|
- 4 KiB code window is real and not advisory. Every fixture is 4491 or 4493
|
|
|
- bytes, so nothing has hit this yet and nothing tests it.
|
|
|
-- **`EmMovAxSp` still emits a 386-only SIB byte** (`8B 44 24 00`). It is correct
|
|
|
- on any 386+ but the SIB byte did not exist in 1984, and the whole premise of
|
|
|
- this project is an 8086. Same class of bug as finding 2 below, unfixed.
|
|
|
+ `rtSz + 1000H` = 4432 and a `.COM` is padded to `max(pc, dc)`, so the fixed
|
|
|
+ 4 KiB code window is real and not advisory. The largest fixture (`t32_forexit`,
|
|
|
+ 135 bytes of code) is nowhere near it, so nothing has hit this and nothing
|
|
|
+ tests it. `rtSz` also *moves* every time the runtime grows, so the window
|
|
|
+ shrinks silently — another derived value that must never be restated.
|
|
|
+- **`[SP]` cannot be encoded on an 8086, and the fix is a shape change.**
|
|
|
+ `EmMovAxSp`/`EmMovCxSp` now emit `POP reg` / `PUSH reg` — two instructions
|
|
|
+ where there used to be one, so anything that assumed a one-instruction
|
|
|
+ emitter has to be revisited. The audit handles them as a documented
|
|
|
+ *sequence*; if another one appears, the sequence machinery is where it goes,
|
|
|
+ not a name hack.
|
|
|
- **String *literals* work; string *variables* do not.** A literal in a
|
|
|
`WRITE`/`WRITELN` argument list is emitted inline and needs no runtime
|
|
|
support beyond `wrtinl`. Declaring `s : string`, assigning to it and
|
|
|
@@ -578,6 +717,14 @@ Details that are deliberate, not incidental:
|
|
|
still said OK.
|
|
|
- **Comma-separated names are not supported.** `var i, c : integer;` is a
|
|
|
parse error. Pre-existing, unrelated to any of the above, and still open.
|
|
|
+- **`readln` of a `BYTE` is a latent 1-byte overflow.** It calls `rdint`, which
|
|
|
+ stores a 2-byte word, so the high byte lands on the next variable. TP3 has a
|
|
|
+ separate `xrdbyte` for exactly this. No fixture declares a `BYTE`, which is
|
|
|
+ the only reason this has never been observed. Write the fixture first.
|
|
|
+- **The program-header parameter loop is unguarded against non-advancing
|
|
|
+ input.** `program p(1;)` would loop forever. The fix needs a `BOOLEAN` flag
|
|
|
+ and **must not** use `EXIT`, which ICEs gm2 in pass 3 (see the pitfalls
|
|
|
+ list). Not yet done.
|
|
|
- **Not implemented** (all `ENoLib`): real, set, record, file, string
|
|
|
*variables*, and any type wider than 2 bytes. `with` is `ENoLib`. `case` *is*
|
|
|
implemented (cascade `CMP`/`JNZ` per label, per RESUME-TP3.md §3.6) but only
|
|
|
@@ -717,6 +864,122 @@ plausible error *number*, (6) a plausible code *size*, and (7) a plausible
|
|
|
nobody runs is documentation. The specific lesson: when two things must
|
|
|
agree, keep one.
|
|
|
|
|
|
+### Then the image actually ran, and eleven more appeared
|
|
|
+
|
|
|
+The sixteen above were found with byte dumps, structural checks and one broken
|
|
|
+emulator. Booting the emitted `.COM` under qemu and comparing **exact output**
|
|
|
+is a different kind of instrument: it catches bugs whose every byte is locally
|
|
|
+correct and whose only defect is that the program goes somewhere else.
|
|
|
+Eleven more, and the two worst in the project are here.
|
|
|
+
|
|
|
+17. **`FOR` was emitted as a post-test loop** — the increment sat outside the
|
|
|
+ body and the test came after it, so the body ran once before the first
|
|
|
+ comparison. TP3's own `emitfor` (TPSRC1, `DoEmit`) tests *before* the body
|
|
|
+ and increments *inside* it. Every `for` fixture printed one line too many,
|
|
|
+ and only `run_com_exec.py` could say so: the byte count was unchanged, the
|
|
|
+ control flow was well-formed, and the matrix stayed green.
|
|
|
+18. **`EXIT` inside a `for` body jumped to the increment, not to the exit.**
|
|
|
+ `t32_forexit` therefore re-tested the condition and could re-enter the body.
|
|
|
+ Worse, the handler had an `EmAddSp (2)` left over from when it exited a
|
|
|
+ `with`-style scope, which unbalanced the argument-cleanup stack. The
|
|
|
+ patch-list sweep also had to be rewritten from
|
|
|
+ `FOR i := a TO exitCnt - 1` to `WHILE i < exitCnt` — `exitCnt` is a
|
|
|
+ `CARDINAL`, so an *empty* range means `TO 65535`, and a program with no
|
|
|
+ `EXIT` in the loop patched 65532 slots. That is a MODULA-2 idiom trap, not
|
|
|
+ a codegen bug.
|
|
|
+19. **The procedure-skip jump was missing, so procedure bodies executed as
|
|
|
+ part of the main body.** The compiler emits a procedure's body *between*
|
|
|
+ the caller's prologue and its own main body, so a jump over it is required.
|
|
|
+ `DeclaresProc ()` now answers "does this program declare any procedure?" by
|
|
|
+ a save/restore-`srcPos` lookahead, and `Compile` emits `EmJmpNear (0)`
|
|
|
+ after the prolog when the answer is TRUE, patching it with
|
|
|
+ `SetPatTgt (overProc, pc)` after `DefPart`. Four fixture code sizes grew by
|
|
|
+ exactly 3 bytes — the `E9` plus its rel16 — and each re-baseline is
|
|
|
+ documented in `expected.tsv` rather than waved through.
|
|
|
+20. **And then the jump's patch slot could not be told from "no jump".**
|
|
|
+ `EmJmpNear (target)` returns a **patch slot index**, and **slot 0 is a
|
|
|
+ legitimate slot**. The sentinel was `overProc := 0`, so
|
|
|
+ `IF overProc # 0 THEN SetPatTgt (...)` *skipped the patch* for a program
|
|
|
+ whose procedure-skip jump happened to be the first patch in the image. The
|
|
|
+ jump kept its placeholder target 0, so it landed at image offset 0 — the
|
|
|
+ entry jump — and the program looped forever, printing nothing. The fix is a
|
|
|
+ separate `hasProc : BOOLEAN`, not a magic slot number.
|
|
|
+ `t31_procparam` *hung* rather than failed, which is how it was found: a
|
|
|
+ fixture that never terminates is a louder signal than one that prints the
|
|
|
+ wrong thing — but only if the harness has a timeout, and only if somebody
|
|
|
+ reads a timeout as information.
|
|
|
+
|
|
|
+ The general lesson is the one this project keeps re-learning: **a
|
|
|
+ plausible placeholder is indistinguishable from a real value.** Slot 0,
|
|
|
+ `0` as "no target", `0` as "no flag set" — each was correct until the
|
|
|
+ first case where the real value was 0. A separate `BOOLEAN` has no
|
|
|
+ collision to have.
|
|
|
+21. **`IF MatchKey (tok) AND (tok = TkElse)` consumed the token it was
|
|
|
+ rejecting.** `AND` is not short-circuit in Modula-2, and a lookahead built
|
|
|
+ out of a *matching* primitive is a parser bug, not a lookahead.
|
|
|
+ `MatchKey` advanced past the keyword, so an `else` that should have been
|
|
|
+ left for the enclosing statement was eaten. The shape that works is
|
|
|
+ `PeekKw (tok)` for the question and `DropB (MatchKey (tok))` for the
|
|
|
+ commit — and `IF <stmt>` may not be the last thing in a compound, which is
|
|
|
+ a separate ISO rule that this hit too.
|
|
|
+22. **A `getbyte` with no pushback slot lost one character per call.** TP3's
|
|
|
+ `getbyte` has a *char pre-read flag*: it reads the next character while
|
|
|
+ looking for digits and then hands it back. The first port had no such slot,
|
|
|
+ so `rdint` consumed the delimiter and the following statement lost its
|
|
|
+ first character. The fix is a one-character pushback slot `D_PUSH` plus
|
|
|
+ `ungetch`, and the slot's *address moves whenever the runtime grows*
|
|
|
+ (`rtSz + LoadBias + dataAt + D_PUSH` — at 432 bytes, image `0x1B0`, memory
|
|
|
+ `0x2B0`, one pad byte before the program header at `0x1B3`). It is computed,
|
|
|
+ never hard-coded: the same class of restated-constant bug as `RT_SZ` above.
|
|
|
+23. **`rdint` did not mirror TP3's `xrdint`/`readnum`.** TP3 checks for `^Z`
|
|
|
+ *first*, then skips characters `<= 20h`, then an optional sign, then digits,
|
|
|
+ then **pushes the terminator back**. Ours skipped leading whitespace only
|
|
|
+ and did not push the terminator back, so two `readln`s misbehaved. `MUL
|
|
|
+ r/m16` also writes `DX`, so the running digit is parked in `DI` — an
|
|
|
+ encoding fact that has to be written down, or the next person
|
|
|
+ "simplifies" it back.
|
|
|
+24. **`INT 21h AH=02h` was given the character in `DL`.** It takes it in `AL`.
|
|
|
+ Four emitters (`MovAlD`, `MovAlSi`, `MovAlArg`, `MovAlDl`) were wrong
|
|
|
+ together, so every character written was the high half of something else —
|
|
|
+ and the *count* of bytes written was right, which is why a byte-counting
|
|
|
+ check passed.
|
|
|
+25. **`EmMovAxSp` / `EmMovCxSp` tried to encode `[SP]`,** which the 8086 has no
|
|
|
+ ModRM form for. They now emit `POP reg` / `PUSH reg` — a two-instruction
|
|
|
+ shape, so the audit treats them as a *sequence* rather than trying to match
|
|
|
+ a mnemonic.
|
|
|
+26. **There is no `[BX]` in mode 16.** `mod=00 rm=111` is `[BX+SI]`, not
|
|
|
+ `[BX]`. A store through a pointer was writing to `BX+SI`. Stores now go
|
|
|
+ through `DI` or `SI`, and absolute data addresses use `mod=00 rm=110` =
|
|
|
+ ModRM `1E`. Worth stating as a rule because the *name* `[BX]` appears in
|
|
|
+ half the 8086 documentation ever written.
|
|
|
+27. **`EmMoveAxDx` emitted `92h` — which is `XCHG AX,DX`.** Behaviour was
|
|
|
+ identical either way, so nothing was broken; only the *name* lied, and a
|
|
|
+ name that lies is how the next one becomes a silent wrong value. It is now
|
|
|
+ `EmXchgAxDx`. The same pass audited the emitter *vocabulary* and renamed
|
|
|
+ the ambiguous ones so the distinction survives: `MovBxImm`/`MovBxVx`
|
|
|
+ (ADDRESS) vs `LdBxVx` (CONTENTS) vs `StVxBx`; `MovAlBl` (`8A C3`, register)
|
|
|
+ vs `LdAlBx` (`8A 07`, memory); `MovAh0` → `{0xB4}` vs `MovAl0` → `{0xB0}`.
|
|
|
+
|
|
|
+## The bug family, stated once
|
|
|
+
|
|
|
+Nine of the twenty-seven are the *same* bug in different clothes: **loading the
|
|
|
+address where the value was wanted, or picking the register one byte or one
|
|
|
+letter away from the right one.** `EmPushVarAddr` had the right bytes for the
|
|
|
+wrong register. `LdAlBx` and `MovAlBl` are one letter apart. `MovAh0` and
|
|
|
+`MovAl0` are one letter apart and their opcodes differ by `04h`. The response
|
|
|
+to that is not more checking — it is that the *names* now carry the
|
|
|
+distinction, so the next one is a compile error instead of a silent wrong
|
|
|
+value. **Check emitted bytes against intended semantics whenever an emitter is
|
|
|
+added**, and let the audit do it for one-liners.
|
|
|
+
|
|
|
+And a second family, on the harness side rather than the compiler's: **a check
|
|
|
+that quietly checks nothing looks exactly like a check that passes.** The audit
|
|
|
+that swept one module, the regex that matched no emitters, the coverage list
|
|
|
+derived from the parser's own output, the four `nonvacuity` mutations that
|
|
|
+matched nothing, and two hard-coded `RT_SZ` constants — six instances, all
|
|
|
+green, all worthless. That is why this file has a non-vacuity section and an
|
|
|
+independently-scanned inventory at all.
|
|
|
+
|
|
|
## gm2 / ISO Modula-2 pitfalls hit along the way
|
|
|
|
|
|
- **Two-phase link** (above) — a single whole-program pass 3 caps
|
|
|
@@ -776,39 +1039,37 @@ plausible error *number*, (6) a plausible code *size*, and (7) a plausible
|
|
|
|
|
|
## Next steps
|
|
|
|
|
|
-1. **Boot a compiled `.COM` and read its output.** This gates everything and
|
|
|
- nothing else can honestly be claimed until it works. The executor is no
|
|
|
- longer the open question — qemu is a proven oracle and the FreeDOS image
|
|
|
- gives a real DOS to run in. Concretely: a boot sector that reads a `.COM`
|
|
|
- off the floppy with `INT 13h` to `0x100`, sets `SS:SP` at the segment top,
|
|
|
- hooks `INT 21h` (`AH=02h/09h/4Ch` → serial), `JMP 0x100`; debug with
|
|
|
- `qemu -d in_asm,exec -D trace.log`; assert the **exact stdout bytes** per
|
|
|
- fixture against expected-output files under `shell/tests/fixtures/`
|
|
|
- (`writeln('hi')` → `hi`). That single assertion is what turns "assembles"
|
|
|
- into "works". Either route works and both are worth having: under FreeDOS
|
|
|
- for realism, under bare qemu with an `INT 21h` shim for reproducibility.
|
|
|
-2. **Re-point `rt_exec.py` at qemu** and require all 33 of its checks to pass.
|
|
|
+1. **Re-point `rt_exec.py` at qemu** and require all 33 of its checks to pass.
|
|
|
They currently fail 33/33 under Unicorn, and those failures are the
|
|
|
- emulator's, not the runtime's. The expectations stay; only the machine
|
|
|
- changes. Add the missing `wrtinl` case, which needs a harness that models
|
|
|
- the caller's contract (length byte and characters at the return address).
|
|
|
-3. **`CmdRun`** as an in-process 8086 interpreter — the `R` menu key, and a
|
|
|
+ emulator's, not the runtime's. **The expectations stay; only the machine
|
|
|
+ changes**, and `tests/exec/bootcom.s` is reused so the boot machinery is
|
|
|
+ written once. This is next because it is the last harness that reports a
|
|
|
+ number nobody can act on, and because the 33 checks are a *per-entry* claim
|
|
|
+ about the runtime, which the 21 execution fixtures only cover end to end.
|
|
|
+ Add the missing `wrtinl` case, which needs a harness that models the
|
|
|
+ caller's contract (length byte and characters at the return address).
|
|
|
+2. **`CmdRun`** as an in-process 8086 interpreter — the `R` menu key, and a
|
|
|
fallback executor for environments with no DOS. Validate it against qemu on
|
|
|
- the *same images*, so the two oracles check each other.
|
|
|
-4. **String *variables*** — `s : string`, `s := 'hi'`, `writeln(s)`. The
|
|
|
+ the *same images*, so the two oracles check each other. Cross-validation is
|
|
|
+ the point: an interpreter that agrees with qemu on 21 fixtures is far more
|
|
|
+ evidence than either alone.
|
|
|
+3. **String *variables*** — `s : string`, `s := 'hi'`, `writeln(s)`. The
|
|
|
encoding blocker is gone (`EmBpDisp`); what is left is a length word, an
|
|
|
- assignment path, and a `WrStr` entry (TPSRC4 `xwrtstr`).
|
|
|
-5. Nested procedures / recursion, `var` parameters (the `SEG:OFF` push from
|
|
|
+ assignment path, and a `WrStr` entry (TPSRC4 `xwrtstr`). `IoCall` currently
|
|
|
+ refuses with `ENoLib`.
|
|
|
+4. Nested procedures / recursion, `var` parameters (the `SEG:OFF` push from
|
|
|
RESUME-TP3.md §3.11), range/index checks (`TU_RANGE_CHECK`,
|
|
|
`TU_INDEX_CHECK`), typed constants (RESUME-TP3.md §3.14), `array` at its
|
|
|
point of use (`t14`), `case` with subrange labels.
|
|
|
-6. **Fix `EmMovAxSp`**, which still emits the 386-only `8B 44 24 00`. On an
|
|
|
- 8086 there is no SIB byte, so the right encoding is `8B 46 00`
|
|
|
- (`MOV AX,[BP+0]`-adjacent form) or a register copy — this needs thinking
|
|
|
- against the *measured* table rather than a habit, and a fixture that reads
|
|
|
- `[SP]`.
|
|
|
-7. Make the 4 KiB code window an enforced limit rather than a documented one:
|
|
|
+5. **`readln` of a `BYTE`** calls `rdint`, which stores 2 bytes and overflows
|
|
|
+ into the next variable. TP3 has a separate `xrdbyte`; a `TU_RdByte` entry is
|
|
|
+ the fix. No fixture exists yet, which is why it has not been done — write
|
|
|
+ the fixture first, so the bug is red before the fix.
|
|
|
+6. Make the 4 KiB code window an enforced limit rather than a documented one:
|
|
|
report an error when `pc` reaches `dc`, instead of writing over the data.
|
|
|
-8. Comma-separated names: `var i, c : integer;`.
|
|
|
-9. Harden the program-header parameter loop against non-advancing input
|
|
|
+7. Comma-separated names: `var i, c : integer;`.
|
|
|
+8. Harden the program-header parameter loop against non-advancing input
|
|
|
(`program p(1;)`) with a `BOOLEAN` flag — **not** `EXIT`, which ICEs gm2.
|
|
|
+9. FreeDOS (`freedos.qcow2`, FD14-LiveCD) is still untried. Not needed for any
|
|
|
+ claim above, but it is the only way to get a *real* DOS as a third opinion
|
|
|
+ on the `INT 21h` shim.
|