v-TP3-8086-LOWERING — Summary ============================= Tag: v-TP3-8086-LOWERING Commit: (the commit that adds this file) What changed ------------ The emitted code no longer contains an opcode the 8086 lacks. `EmJcc` emitted `0F 8x rel16` and `EmSetcc` emitted `0F 9x rel8` (SETcc). `0F` is a 386-and-later opcode-escape prefix; the 8086 — the machine TP3 targets, and the machine this compiler exists to emit for — has none. Every relational operator and every conditional branch in the compiler went through these two emitters, so this was not two call sites but the whole idiom, and the output was not wrong code: it was not code. Replacement, read off the disassembly of the original rather than invented: TPSRC8:247-252 if CALL excond ; MOV AL,brnchop ; MOV AH,#$03 ; TPSRC8:271-276 while (same shape) TPSRC8:292-297 repeat (same shape, jumping back to the loop head) -> a SHORT Jcc of displacement 3, stepping over a 3-byte EJMP TPSRC9:412-424 flgbool, emitting at 418-422 CALL ecode ; B $03,$B8,$01,$00 * MOV AX,#0001 MOV AL,brnchop CALL ebyte CALL ecode * offset B $02,$01,$48 * DEC AX -> AX is 1 if the branch is taken, 0 if it ran Both are 8086 code and both are shorter than what they replace. The condition rides in the opcode's **low nibble** — `70H + cc` carries the identical condition `0F 8x` / `0F 9x` did — so all seven `EmJcc` sites and all six `EmSetcc` arms go on passing the byte they always passed, unchanged. The flags survive, which the `FOR` test needs: it emits `CMP` then `Jcc` with nothing in between. `JccShort` and `JccShortInv` are two named helpers rather than open-coded expressions, because an open-coded negation at two call sites is how they end up disagreeing. `JccShortInv` spells `n XOR 1` as `n + 1 - 2 * (n MOD 2)`: gm2 under `-fiso` rejects `BITAND` and `BAND` as syntax errors and rejects arithmetic on a `BYTE` operand, so there is no XOR at all and every operand goes through `VAL`. Bugs found and fixed -------------------- 32. **`0F 8x` / `0F 9x` throughout — an illegal instruction on the target.** Present since the code generator was written, with everything green. 33. **The first attempt at that port inverted every conditional in every program.** `EmJcc` jumps *to* its target, but the 8086 shape steps *over* an `EJMP`, so writing `7x 03` and falling into the destination runs the two the wrong way round. The polarity is genuinely opposite: TP3's `brnchop` is taken when the condition is TRUE (`TPSRC9:35` sets it to `#$75`, JNZ, "AX is non-zero"; `TPSRC9:90` loads the comparison table's own opcode for the flags-set case), whereas the nibbles these call sites pass are taken when the condition is **FALSE** — `IF`'s `EmJcc (84H)` is `JZ` patched to the `ELSE`, so it must fire when the test failed. `EmSetcc` therefore keeps the nibble as it stands and `EmJcc` negates it. `t09_if` printed `pos / nonpos / lt` for a program whose hand-derived output is `nonpos / pos / ge`, and `t12_repeat` looped forever. Only the execution suite noticed — third instance in a row of a fault that passed every byte-level check in this project. Tests / harness --------------- - `run_all.sh`: 12 checks ALL PASS, up from 11; the new one is `check_8086.py`. - `nonvacuity.sh`: 44 ok / 0 failed, up from 39. - Compile matrix: 34 passed / 0 failed. - Execution: 31 passed / 0 failed. - .COM linker: 31 checked / 0 failed. - Runtime entries: 36 passed / 0 failed. - `check_8086.py`: 26 comparison sites, 22 lowered to a Boolean value, 13 to a branch; all 6 declared comparison conditions and all 4 declared branch conditions exercised; 0 `0F`-prefixed opcodes in the runtime (219 bytes swept) or in swept program code (28 of 31 fixtures, 2257 bytes — the other 3 have inline strings, which is printed rather than implied). `tests/check_8086.py` is new, and its design is mostly a list of what does **not** work: - A **linear sweep of the code region is not a sound oracle.** Inline string literals are emitted into the code stream, so the sweep desynchronises and then aborts on text: `t09_if` decodes 16 real instructions and dies on `FE E9 0D 00`, which is ASCII. - A **bare `3D` anchor is unsound.** It matches displacement and immediate bytes as readily as opcodes, and using one produced two false alarms — a `3D` inside a `CALL` displacement in `t11_for`, and one inside a string in `t21_mixed`. The anchor is the three-byte `3D 00 00` or nothing. So it anchors on **comparison sites** rather than instruction boundaries: `3B C1` (`EmCmpAxCx`) and `3D 00 00` (`EmCmpAxi (0)`) are the only sequences that can precede a lowering. Branches are *additionally* found by shape (`7x 03 E9`, searching for the `E9`), which is what covers the `CASE` arm whose `EmCmpAxi` carries a label rather than 0. "Is this an 8086 shape" is a much weaker question than it looks — a `SETG` where a `SETGE` belongs is still a fine shape — so two clauses compare against **source**: - **G** pins `t33_cmpops`'s 13 comparisons against the operators in source order. It catches a `>` / `>=` swap, and the failure message shows the swap (`… Dh Dh Fh Fh Dh` where the source asks `… Fh Fh Dh Dh Fh`). - **H** pins, per fixture, the multiset of conditions its branch sites declare, read off the `.pas` sources with the reasoning written beside each row. **H's need was measured, not anticipated.** Mutation M5 — the polarity inversion applied to the IF and CASE sites only — left the check **green** while every conditional in every program took the wrong path, because `IF` declares nibble 4, `CASE` declares 5, and they negate into each other. The whole-suite version (M4) was caught only by luck: `FOR` declares `C` and `F`, whose negations `D` and `E` are declared for nothing at all. M5's first run coming back green is the entire reason H exists. Five new non-vacuity cases, each run by hand and confirmed red before being written down: M1 EmJcc back to `0F 8x rel16` -> red, 45 problems M2 EmSetcc back to `0F 9x` + MOV AH,0 -> red, 49 problems M3 the `>` / `>=` SETcc arms swapped (the old bug 29) -> red, 2 problems M4 branch polarity inverted everywhere (bug 33) -> red, 22 problems M5 branch polarity inverted for IF and CASE only -> red, 12 problems Two harness faults were found while building them, both of which had produced false results before: - The 8086 group's first draft rebuilt with `make`, which does **not** build `comtest`, so `check_8086.py` linked the stale compiler and M1 came back **VACUOUS**. A helper that exits non-zero makes `if mutate_x; then` false, so the case is silently *skipped* — which looks exactly like a pass. The helpers now check the rebuild and say so. - A first cross-check of the size re-baseline reported "no mismatch" because no `.COM` files existed where it looked, so its loop never ran and a vacuous "all clear" came back. It was re-run with an assertion on the image count, and this time reported the real answer. Worth recording because the failure looked exactly like success — but note that cross-check was a throwaway command, **not** a committed script, so the guard does not guard anything permanent. `run_compile_tests.sh` re-baselining is still a manual edit. Scope decisions and findings ---------------------------- - `t33_cmpops` is a new fixture, and it exists partly for clause G: `=`, `<>` and `<=` had **never been used as a comparison anywhere in the suite**, and `>` / `>=` had already been swapped once, so the coverage hole and the bug it let through were the same hole. Its `.out` is hand-derived from the Pascal and was written **before** the machine ran. - `expected.tsv` re-baselined for nine control-flow fixtures, every one upward by exactly **+1 per lowered site** (a branch 4 -> 5 bytes, a value 5 -> 6). The site counts were counted out of the linked images and each row written only after its delta matched its own count; the arithmetic is spelled out in the file. Data sizes did not move. Byte-level expectations for those sites now live in `check_8086.py`, behaviour in `run_com_exec.py`; the matrix row only says how big the code is. - `check_8086.py` cannot assert on the `CASE` arm's comparison: it is `3D lo hi` with a non-zero label immediate, and no sound anchor can find it. The arm is covered by shape and by `t22_case`'s execution, but the label immediate itself is unchecked. - The check executes nothing. It proves the opcodes are 8086 and that conditions are attached to the right constructs; the control flow those bytes produce is still only checked by qemu **on a 486**. Tag placement ------------- `v-TP3-8086-LOWERING` points at the commit that adds this file, so its tree contains this milestone's account of it — the convention every other tag follows, and the one `v-TP3-COM-IMAGE` departs from for reasons recorded in its own tag message. (The tag was first placed on the work commit and moved once this file was written; both commits are in the tree, and nothing is pushed.) What remains (next steps) ------------------------- 1. `CmdRun` (`R` key): in-process 8086 interpreter, cross-validated against qemu on the same images. Also the **only** candidate for a real 8086 execution oracle, since qemu cannot be one. 2. String variables (`s : string`, assignment, `writeln(s)`). 3. Nested procedures/recursion, `var` parameters (`SEG:OFF`), range/index checks, typed constants, array at use (`t14`), case subrange labels. 4. `readln` of a `BYTE` (separate `xrdbyte` / `TU_RdByte`). 5. Enforce the 4 KiB code window (error if `pc` reaches data start). 6. Harmonise multi-name declarations (both `,` and `;` spellings). 7. Harden program-header parameter loop against non-advancing input. 8. FreeDOS as a third opinion (untried).