|
@@ -16,7 +16,8 @@ manual, not guessed.
|
|
|
| Compiler skeleton + build recipe | `v-TP3-SHELL-COMPILES` | done |
|
|
| Compiler skeleton + build recipe | `v-TP3-SHELL-COMPILES` | done |
|
|
|
| Parser `Skip` bug class (9 sites) | `v-TP3-PARSER-FIXES` | done |
|
|
| Parser `Skip` bug class (9 sites) | `v-TP3-PARSER-FIXES` | done |
|
|
|
| Standard procedures + `rel16` fix | `v-TP3-STDPROCS` | done |
|
|
| Standard procedures + `rel16` fix | `v-TP3-STDPROCS` | done |
|
|
|
-| Runtime blob, linker, `CmdRun` interpreter | — | **not started** |
|
|
|
|
|
|
|
+| Runtime library + 8086 execution harness | `v-TP3-RUNTIME-BLOB` | assembled, **never run** |
|
|
|
|
|
+| Linker + `CmdRun` | — | **not started** |
|
|
|
|
|
|
|
|
## Build
|
|
## Build
|
|
|
|
|
|
|
@@ -93,6 +94,62 @@ changing `Run`'s signature, so `Editor.def` stays additive.
|
|
|
feature: the latter move as the compiler grows, and a test whose
|
|
feature: the latter move as the compiler grows, and a test whose
|
|
|
expectations drift with it stops being a test.
|
|
expectations drift with it stops being a test.
|
|
|
|
|
|
|
|
|
|
+## The executor problem (blocking everything downstream)
|
|
|
|
|
+
|
|
|
|
|
+The whole point of a Pascal→8086 compiler is that the output *runs*. Until this
|
|
|
|
|
+milestone no compiled image had ever been executed, so the plan was: get a real
|
|
|
|
|
+CPU emulator, run the image, assert the exact stdout bytes.
|
|
|
|
|
+
|
|
|
|
|
+**Unicorn 2.1.4 cannot be used for this.** `UC_MODE_16` mis-decodes 16-bit
|
|
|
|
|
+ModRM memory operands. The measurement, by loading a byte-pattern image so a
|
|
|
|
|
+load reveals its own effective address, then executing a single
|
|
|
|
|
+`LEA AX,[r+disp8]` and reading AX back with every register set to a distinct
|
|
|
|
|
+value:
|
|
|
|
|
+
|
|
|
|
|
+```
|
|
|
|
|
+mod=01, disp8=4 got 8086 says
|
|
|
|
|
+ 8D 40 04 LEA AX,[BX+4] AX=0d04 0x504 (= BX+SI+4) WRONG
|
|
|
|
|
+ 8D 41 04 LEA AX,[BX+SI+4] AX=0e04 0xd04 WRONG
|
|
|
|
|
+ 8D 42 04 LEA AX,[BX+DI+4] AX=0f04 0xe04 WRONG
|
|
|
|
|
+ 8D 43 04 LEA AX,[BP+4] AX=1004 0x704 WRONG
|
|
|
|
|
+ 8D 45 04 LEA AX,[DI+4] AX=0904 0x904 right
|
|
|
|
|
+ 8D 46 04 LEA AX,[BP+4] AX=0704 0x704 right
|
|
|
|
|
+ 8D 47 04 LEA AX,[DI+4] AX=0504 0x904 WRONG
|
|
|
|
|
+mod=00 / mod=10 direct disp16
|
|
|
|
|
+ 8B 1E 00 20 MOV BX,[2000] BX=1234 right
|
|
|
|
|
+ 8D 06 34 12 LEA AX,[1234] AX=1234 right
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+So `rm=5` and `rm=6` decode correctly but `rm=0,1,2,3,7` do not, and only
|
|
|
|
|
+*base-register-free* addressing (direct `disp16`) is trustworthy. That rules
|
|
|
|
|
+Unicorn out as an oracle for exactly the instruction forms generated code is
|
|
|
|
|
+made of — `LEA AX,[BP+d]`, `MOV AX,[SI+d]`, `LODSW`-style loops, everything
|
|
|
|
|
+with a frame pointer. It cannot distinguish "my codegen is wrong" from "the
|
|
|
|
|
+emulator is wrong", which makes it worse than no emulator at all.
|
|
|
|
|
+
|
|
|
|
|
+`pip install --upgrade unicorn` resolves to the same 2.1.4, so this is not
|
|
|
|
|
+avoidable by upgrading.
|
|
|
|
|
+
|
|
|
|
|
+Also ruled out, for the record:
|
|
|
|
|
+
|
|
|
|
|
+- **DOSBox-X 2025.02.01** (installed, `/usr/bin/dosbox-x`, plain root-owned
|
|
|
|
|
+ ELF, not a snap) starts cleanly headless under
|
|
|
|
|
+ `SDL_VIDEODRIVER=dummy SDL_AUDIODRIVER=dummy`, but its `-c`/autoexec
|
|
|
|
|
+ commands never observably execute: a `md` never appeared on the host, and a
|
|
|
|
|
+ `.COM` that creates `OUT.TXT` via `INT 21h AH=3Dh/40h` never produced the
|
|
|
|
|
+ file. Shell `>` is intercepted by dosbox-x's own wrapper
|
|
|
|
|
+ (`SHELL:Redirect output to out.txt`). A `.BAT` route timed out.
|
|
|
|
|
+- **qemu-system-i386 is installed** (`/usr/bin/qemu-system-i386`) and is the
|
|
|
|
|
+ next candidate: a 512-byte boot sector can load the image and a `INT 21h`
|
|
|
|
|
+ shim can hand the output back. Not attempted yet.
|
|
|
|
|
+
|
|
|
|
|
+`tests/rt_exec.py` is the harness, written against Unicorn, and it is written
|
|
|
|
|
+to survive the switch: it loads the runtime, calls each entry with a known
|
|
|
|
|
+argument, and compares the bytes sent to `INT 21h` against expectations. It
|
|
|
|
|
+currently **fails**, and the failures are the emulator's, not the library's —
|
|
|
|
|
+`wrint` prints `-` for every value because `MOV AX,[BP+4]` reads the wrong
|
|
|
|
|
+address. Do not read those results as a verdict on the runtime.
|
|
|
|
|
+
|
|
|
## Components
|
|
## Components
|
|
|
|
|
|
|
|
### Shell — `shell/Shell.mod`, `Term.mod`, `Posix.c`, `TextBuf.mod`
|
|
### Shell — `shell/Shell.mod`, `Term.mod`, `Posix.c`, `TextBuf.mod`
|
|
@@ -161,17 +218,83 @@ image-base space (`TU_InitMem=8H`, `TU_ProgEnd=10H`, `TU_StackChk=18H`).
|
|
|
|
|
|
|
|
Detail: `TP3-COMPILER.md`.
|
|
Detail: `TP3-COMPILER.md`.
|
|
|
|
|
|
|
|
|
|
+### Runtime library — `shell/Runtime.mod`, `shell/Runtime.def`
|
|
|
|
|
+
|
|
|
|
|
+The 8086 runtime, **assembled byte by byte from Modula-2** — no external
|
|
|
|
|
+assembler and no checked-in binary, so the tree stays self-contained and the
|
|
|
|
|
+entry offsets are *derived* rather than guessed. Each emitter is one
|
|
|
|
|
+instruction with its ModRM byte spelled out in a comment so the encoding can
|
|
|
|
|
+be checked by hand against an 8086 table. This is the same approach
|
|
|
|
|
+`Compiler.mod` already takes (`Ebyte`/`Eword`/`EmCall`).
|
|
|
|
|
+
|
|
|
|
|
+It mirrors the original's own mechanism: TPSRC7 `copyrt` copies the runtime
|
|
|
|
|
+into the front of the code buffer (`SI=DI=0`, `REPZ MOVSB`) and `pc` is then
|
|
|
|
|
+initialised past it (`MOV pc,#$2D7C`). Same shape here — the compiler is meant
|
|
|
|
|
+to copy the blob to the front of `cbuf` and start `pc`/`dc` past it. Runtime
|
|
|
|
|
+data therefore lives at fixed low offsets and needs no relocation, and because
|
|
|
|
|
+both sides of every `CALL` shift by the same amount, `EmCall`'s displacement
|
|
|
|
|
+arithmetic is unaffected by the runtime being prepended.
|
|
|
|
|
+
|
|
|
|
|
+Current blob: **366 bytes**, 13 entries, offsets read back out of the
|
|
|
|
|
+assembled bytes by `tests/RtProbe.mod`:
|
|
|
|
|
+
|
|
|
|
|
+| entry | offset | entry | offset | entry | offset |
|
|
|
|
|
+|---|---|---|---|---|---|
|
|
|
|
|
+| `initmem` | 0 | `wrint` | 36 | `rdint` | 142 |
|
|
|
|
|
+| `progend` | 28 | `wrchar` | 97 | `rdchar` | 243 |
|
|
|
|
|
+| `stackchk` | 35 | `wrbool` | 106 | `rdbool` | 264 |
|
|
|
|
|
+| `halt` | 28 | `wrreal` | 126 | `rdln` | 310 |
|
|
|
|
|
+| | | `wrln` | 134 | | |
|
|
|
|
|
+
|
|
|
|
|
+`progend` and `halt` deliberately share one address (`XOR AX,AX / MOV AH,4C /
|
|
|
|
|
+INT 21h / RET`): the compiler already zeroes AX before `progend` and discards
|
|
|
|
|
+the `HALT` argument at compile time, so both leave with exit code 0.
|
|
|
|
|
+
|
|
|
|
|
+Conventions, matching `Compiler.IoCall` exactly:
|
|
|
|
|
+
|
|
|
|
|
+| entry | argument | notes |
|
|
|
|
|
+|---|---|---|
|
|
|
|
|
+| `WrInt/WrChar/WrBool/WrReal` | one 16-bit **value** on the stack | caller pops |
|
|
|
|
|
+| `RdInt/RdChar/RdBool` | one **address** on the stack | caller pops |
|
|
|
|
|
+| `WrLn/RdLn/StackChk` | nothing | |
|
|
|
|
|
+| `InitMem` | `AX` = offset of the program header | a *register*, not a stack word |
|
|
|
|
|
+| `ProgEnd/Halt` | nothing | exits, code 0 |
|
|
|
|
|
+
|
|
|
|
|
+`InitMem` reads the data base and end out of the header words at `+2`/`+6` and
|
|
|
|
|
+zeroes that range, because Pascal leaves globals undefined. `StackChk` is a
|
|
|
|
|
+bare `RET` — range and stack checking aren't compiled in yet, and the call site
|
|
|
|
|
+sits mid-expression, so it must not touch a register.
|
|
|
|
|
+
|
|
|
|
|
+Two label-name plus fixup list: `rel8`, `rel16` and runtime-data addresses are
|
|
|
|
|
+all patched after the blob is placed, so nothing depends on a hand-computed
|
|
|
|
|
+displacement.
|
|
|
|
|
+
|
|
|
|
|
+**It has never executed.** See the emulator finding below.
|
|
|
|
|
+
|
|
|
|
|
+
|
|
|
## Honest limitations
|
|
## Honest limitations
|
|
|
|
|
|
|
|
-- **A compiled image still cannot be executed.** `CmdRun` is a stub and the
|
|
|
|
|
- runtime blob does not exist. The emitted code is verified *byte by byte*
|
|
|
|
|
- against the offsets the compiler intends, but nothing has ever run it.
|
|
|
|
|
-- **The `TU_*` entry offsets are placeholders under an assumed model** — a
|
|
|
|
|
- runtime blob prepended to the image, with every `pc` rebased by its size.
|
|
|
|
|
- That model is assumed, not implemented; the linker is unwritten. Note the
|
|
|
|
|
|
|
+- **A compiled image still cannot be executed.** `CmdRun` is a stub, the
|
|
|
|
|
+ linker is unwritten, and no 8086 executor on this machine has yet proved
|
|
|
|
|
+ trustworthy (above). The emitted code is verified *byte by byte* against the
|
|
|
|
|
+ offsets the compiler intends, but nothing has ever run it.
|
|
|
|
|
+- **The runtime is written but unproven.** `Runtime.mod` assembles to 366 bytes
|
|
|
|
|
+ and its entry offsets are derived from the emitted bytes, but it has never
|
|
|
|
|
+ been executed on a correct CPU, so treat every encoding in it as unverified
|
|
|
|
|
+ even though three of its own bugs were caught by running it under a broken
|
|
|
|
|
+ emulator.
|
|
|
|
|
+- **The compiler is not yet wired to the runtime.** `Compiler.mod` still
|
|
|
|
|
+ carries the hardcoded placeholder `TU_*` constants (`TU_InitMem=8H`,
|
|
|
|
|
+ `TU_ProgEnd=10H`, …) and still starts `pc` at 0, so emitted images do not
|
|
|
|
|
+ contain the runtime and those offsets are still wrong. `Runtime.mod` is not
|
|
|
|
|
+ in `make` or `run_compile_tests.sh` yet for the same reason. Note the
|
|
|
prologue's `CALL TU_InitMem` targets offset 8, which is currently the
|
|
prologue's `CALL TU_InitMem` targets offset 8, which is currently the
|
|
|
- `hdrMax` header word, so the scheme is only coherent once the runtime is
|
|
|
|
|
- actually prepended.
|
|
|
|
|
|
|
+ `hdrMax` header word — coherent only once the blob is really prepended.
|
|
|
|
|
+ *(Correction to the earlier note in this file: the `TU_InitMem=8` "collision"
|
|
|
|
|
+ was a false alarm. Per TPSRC7 the runtime is copied to the *front* of the
|
|
|
|
|
+ code buffer and `pc` starts past it, so `TU_*` offsets are runtime-relative,
|
|
|
|
|
+ not image-absolute, and no rebasing of the displacement arithmetic is
|
|
|
|
|
+ needed.)*
|
|
|
- **No string runtime.** `RdConst` gives a 1-character literal as `TScalar`
|
|
- **No string runtime.** `RdConst` gives a 1-character literal as `TScalar`
|
|
|
(its char code) and only longer literals as `TString`, so multi-char
|
|
(its char code) and only longer literals as `TString`, so multi-char
|
|
|
literals raise `ENoLib`. `writeln('a')` works via a `chr` flag on `ERes`;
|
|
literals raise `ENoLib`. `writeln('a')` works via a `chr` flag on `ERes`;
|
|
@@ -211,6 +334,33 @@ hand — code sizes looked perfectly plausible throughout.
|
|
|
too large (`0010` shown as `00000100`). A misleading tool is worse than
|
|
too large (`0010` shown as `00000100`). A misleading tool is worse than
|
|
|
none — it corrupts any offset arithmetic done from its output.
|
|
none — it corrupts any offset arithmetic done from its output.
|
|
|
|
|
|
|
|
|
|
+## Bugs found by actually running code
|
|
|
|
|
+
|
|
|
|
|
+Executing the hand-assembled runtime — even under a broken emulator — paid for
|
|
|
|
|
+itself immediately, because a wrong encoding *executes* rather than failing to
|
|
|
|
|
+assemble. Four, none of which a compiler diagnostic would ever have reported.
|
|
|
|
|
+
|
|
|
|
|
+1. **`B()` silently truncated multi-byte opcodes.** `PROCEDURE B` emits exactly
|
|
|
|
|
+ one byte and masks with `MOD 100H`, so `B (8BE4H)` — a two-byte opcode passed
|
|
|
|
|
+ as one literal — emitted just `E4`. `MOV BP,SP` was missing from every frame
|
|
|
|
|
+ in the runtime, so `BP` stayed 0 and *every* `BP`-relative access read
|
|
|
|
|
+ address 4. Found by decoding the hex dump; the byte is gone, not wrong.
|
|
|
|
|
+2. **`MOV BP,SP` encoded as `8B E4`, which is `MOV SP,SP`** — a no-op. The
|
|
|
|
|
+ ModRM byte is `mod·64 + reg·8 + rm`, and I mis-derived it. `Compiler.mod`'s
|
|
|
|
|
+ `EmMovBpSp` had it right all along (`8B 0CH`); only the new module was wrong.
|
|
|
|
|
+ This is why the encoding is now written as three explicit `B` calls with the
|
|
|
|
|
+ arithmetic in a comment rather than as one hex literal.
|
|
|
|
|
+3. **`InitMem` read its argument from `[SP]`** — the return address. The
|
|
|
|
|
+ convention is a register (`AX`), unlike the per-argument I/O entries which
|
|
|
|
|
+ do take a stack word. Caught because the data area was never cleared.
|
|
|
|
|
+4. **`EmPushVarAddr` computes the wrong base register for locals** (pre-existing,
|
|
|
|
|
+ **not yet fixed**). It emits `8D 46 disp`, which is `LEA AX,[SI+disp8]`, but
|
|
|
|
|
+ intends `LEA AX,[BP+disp8]` = `8D 45 disp`. So `read` into a *local* variable
|
|
|
|
|
+ has always addressed the wrong cell, silently. It also truncates the offset
|
|
|
|
|
+ with `off MOD 100H`, losing displacements above 255. The global form
|
|
|
|
|
+ `8D 06 off` (`LEA AX,[disp16]`) is correct. Found while writing the runtime's
|
|
|
|
|
+ read entries, which needed the same encoding to be right.
|
|
|
|
|
+
|
|
|
## gm2 / ISO Modula-2 pitfalls hit along the way
|
|
## gm2 / ISO Modula-2 pitfalls hit along the way
|
|
|
|
|
|
|
|
- **Two-phase link** (above) — a single whole-program pass 3 caps
|
|
- **Two-phase link** (above) — a single whole-program pass 3 caps
|
|
@@ -222,6 +372,10 @@ hand — code sizes looked perfectly plausible throughout.
|
|
|
- A bare `HALT` aborts under `-fiso` (SIGABRT, exit 134); `HALT (0)` is
|
|
- A bare `HALT` aborts under `-fiso` (SIGABRT, exit 134); `HALT (0)` is
|
|
|
correct.
|
|
correct.
|
|
|
- `CHAR` is not the ZType: `ch = 09H` must be `ORD (ch) = 09H`.
|
|
- `CHAR` is not the ZType: `ch = 09H` must be `ORD (ch) = 09H`.
|
|
|
|
|
+- gm2's ISO `SYSTEM` exports `ORD` but **not** `Ord` — the import is
|
|
|
|
|
+ case-sensitive here despite gm2's usual case-insensitivity, so
|
|
|
|
|
+ `FROM SYSTEM IMPORT Ord` fails with "unknown symbol" while plain `ORD (c)`
|
|
|
|
|
+ compiles. Write `ORD`, never `Ord`.
|
|
|
- ISO forbids dropping a function result in a statement → the `DropCh`/
|
|
- ISO forbids dropping a function result in a statement → the `DropCh`/
|
|
|
`DropB`/`DropC` discard helpers wrap ~39 call sites.
|
|
`DropB`/`DropC` discard helpers wrap ~39 call sites.
|
|
|
- `AND`/`OR`/`NOT` on 16-bit `CARDINAL` → `BitAnd`/`BitOr`/`BitNot`.
|
|
- `AND`/`OR`/`NOT` on 16-bit `CARDINAL` → `BitAnd`/`BitOr`/`BitNot`.
|
|
@@ -253,11 +407,28 @@ hand — code sizes looked perfectly plausible throughout.
|
|
|
|
|
|
|
|
## Next steps
|
|
## Next steps
|
|
|
|
|
|
|
|
-1. **Runtime blob + linker.** Everything downstream is gated on it: the
|
|
|
|
|
- `TU_*` entries have to land somewhere real and every `pc` needs rebasing.
|
|
|
|
|
-2. **`CmdRun`** — the interpreter. Without it no compiled program has ever
|
|
|
|
|
- run, so the codegen is unproven in the only way that counts.
|
|
|
|
|
-3. **String runtime** — unlocks the last 4 real fixture failures.
|
|
|
|
|
-4. Nested procedures / recursion, `var` parameters (the `SEG:OFF` push from
|
|
|
|
|
|
|
+1. **Get a trustworthy 8086 executor**, because it gates items 2–4 and nothing
|
|
|
|
|
+ else can be claimed until a compiled image runs. `qemu-system-i386` with a
|
|
|
|
|
+ boot-sector loader and an `INT 21h` shim is the plan; keep `rt_exec.py`'s
|
|
|
|
|
+ expectations and change only the machine behind them. (Do *not* reach for
|
|
|
|
|
+ Unicorn's 16-bit mode again, and do not re-derive the `rm` table by hand
|
|
|
|
|
+ again — check it against a real decoder.)
|
|
|
|
|
+2. **Fix `EmPushVarAddr`** — `8D 45`/`8D 85` for locals, full `disp16`, per
|
|
|
|
|
+ bug 4 above. Two lines, and `read` into a local is wrong until it is done.
|
|
|
|
|
+3. **Wire the runtime in and write the linker**: `pc := RT_Size`,
|
|
|
|
|
+ `dc := RT_Size + 1000H` (a fixed 4 KiB code/data gap, so a program's data
|
|
|
|
|
+ can't collide with its code in a single 64 K `.COM` segment), take the
|
|
|
|
|
+ `TU_*` offsets from `RT_Entry` instead of the hardcoded constants, patch the
|
|
|
|
|
+ header words (`hdrDS` = data base, `hdrHeap` = data end, so `InitMem` can
|
|
|
|
|
+ zero globals), pad the image to cover the data area, and emit the `.COM`.
|
|
|
|
|
+ Add `Runtime` to the `make` and `run_compile_tests.sh` rebuild lists.
|
|
|
|
|
+4. **Prove it end to end**: compile a fixture, link, execute, assert the exact
|
|
|
|
|
+ stdout bytes (`writeln('hi')` → `hi`). That single assertion is what turns
|
|
|
|
|
+ this from "assembles" into "works".
|
|
|
|
|
+5. **`CmdRun`** — run the emitted image from the `R` menu key.
|
|
|
|
|
+6. **String runtime** — unlocks the last 4 real fixture failures.
|
|
|
|
|
+7. Nested procedures / recursion, `var` parameters (the `SEG:OFF` push from
|
|
|
RESUME-TP3.md §3.11), range/index checks (`TU_RANGE_CHECK`,
|
|
RESUME-TP3.md §3.11), range/index checks (`TU_RANGE_CHECK`,
|
|
|
- `TU_INDEX_CHECK`), and typed constants (RESUME-TP3.md §3.14).
|
|
|
|
|
|
|
+ `TU_INDEX_CHECK`), typed constants (RESUME-TP3.md §3.14).
|
|
|
|
|
+8. Harden the program-header parameter loop against non-advancing input
|
|
|
|
|
+ (`program p(1;)`) with a `BOOLEAN` flag — **not** `EXIT`, which ICEs gm2.
|