TP3/v-TP3-8086-LOWERING.md

v-TP3-8086-LOWERING — Summary

Tag: v-TP3-8086-LOWERING Commit: (the commit that adds this file)

What changed

The emitted code no longer contains an opcode the 8086 lacks.

EmJcc emitted 0F 8x rel16 and EmSetcc emitted 0F 9x rel8 (SETcc). 0F is a 386-and-later opcode-escape prefix; the 8086 — the machine TP3 targets, and the machine this compiler exists to emit for — has none. Every relational operator and every conditional branch in the compiler went through these two emitters, so this was not two call sites but the whole idiom, and the output was not wrong code: it was not code.

Replacement, read off the disassembly of the original rather than invented:

TPSRC8:247-252  if     CALL excond ; MOV AL,brnchop ; MOV AH,#$03 ;
TPSRC8:271-276  while  (same shape)
TPSRC8:292-297  repeat (same shape, jumping back to the loop head)
    -> a SHORT Jcc of displacement 3, stepping over a 3-byte EJMP

TPSRC9:412-424  flgbool, emitting at 418-422
    CALL ecode ; B $03,$B8,$01,$00   * MOV AX,#0001
    MOV  AL,brnchop
    CALL ebyte
    CALL ecode                       * offset
    B    $02,$01,$48                 * DEC AX
    -> AX is 1 if the branch is taken, 0 if it ran

Both are 8086 code and both are shorter than what they replace. The condition rides in the opcode's low nibble — 70H + cc carries the identical condition 0F 8x / 0F 9x did — so all seven EmJcc sites and all six EmSetcc arms go on passing the byte they always passed, unchanged. The flags survive, which the FOR test needs: it emits CMP then Jcc with nothing in between.

JccShort and JccShortInv are two named helpers rather than open-coded expressions, because an open-coded negation at two call sites is how they end up disagreeing. JccShortInv spells n XOR 1 as n + 1 - 2 * (n MOD 2): gm2 under -fiso rejects BITAND and BAND as syntax errors and rejects arithmetic on a BYTE operand, so there is no XOR at all and every operand goes through VAL.

Bugs found and fixed

1. 0F 8x / 0F 9x throughout — an illegal instruction on the target. Present since the code generator was written, with everything green.

2. The first attempt at that port inverted every conditional in every program. EmJcc jumps to its target, but the 8086 shape steps over an EJMP, so writing 7x 03 and falling into the destination runs the two the wrong way round. The polarity is genuinely opposite: TP3's brnchop is taken when the condition is TRUE (TPSRC9:35 sets it to #$75, JNZ, "AX is non-zero"; TPSRC9:90 loads the comparison table's own opcode for the flags-set case), whereas the nibbles these call sites pass are taken when the condition is FALSE — IF's EmJcc (84H) is JZ patched to the ELSE, so it must fire when the test failed. EmSetcc therefore keeps the nibble as it stands and EmJcc negates it.

   t09_if printed pos / nonpos / lt for a program whose hand-derived output is nonpos / pos / ge, and t12_repeat looped forever. Only the execution suite noticed — third instance in a row of a fault that passed every byte-level check in this project.

Tests / harness

• run_all.sh: 12 checks ALL PASS, up from 11; the new one is check_8086.py.
• nonvacuity.sh: 44 ok / 0 failed, up from 39.
• Compile matrix: 34 passed / 0 failed.
• Execution: 31 passed / 0 failed.
• .COM linker: 31 checked / 0 failed.
• Runtime entries: 36 passed / 0 failed.
• check_8086.py: 26 comparison sites, 22 lowered to a Boolean value, 13 to a branch; all 6 declared comparison conditions and all 4 declared branch conditions exercised; 0 0F-prefixed opcodes in the runtime (219 bytes swept) or in swept program code (28 of 31 fixtures, 2257 bytes — the other 3 have inline strings, which is printed rather than implied).

tests/check_8086.py is new, and its design is mostly a list of what does not work:

• A linear sweep of the code region is not a sound oracle. Inline string literals are emitted into the code stream, so the sweep desynchronises and then aborts on text: t09_if decodes 16 real instructions and dies on FE E9 0D 00, which is ASCII.
• A bare 3D anchor is unsound. It matches displacement and immediate bytes as readily as opcodes, and using one produced two false alarms — a 3D inside a CALL displacement in t11_for, and one inside a string in t21_mixed. The anchor is the three-byte 3D 00 00 or nothing.

So it anchors on comparison sites rather than instruction boundaries: 3B C1 (EmCmpAxCx) and 3D 00 00 (EmCmpAxi (0)) are the only sequences that can precede a lowering. Branches are additionally found by shape (7x 03 E9, searching for the E9), which is what covers the CASE arm whose EmCmpAxi carries a label rather than 0.

"Is this an 8086 shape" is a much weaker question than it looks — a SETG where a SETGE belongs is still a fine shape — so two clauses compare against source:

• G pins t33_cmpops's 13 comparisons against the operators in source order. It catches a > / >= swap, and the failure message shows the swap (… Dh Dh Fh Fh Dh where the source asks … Fh Fh Dh Dh Fh).
• H pins, per fixture, the multiset of conditions its branch sites declare, read off the .pas sources with the reasoning written beside each row.

H's need was measured, not anticipated. Mutation M5 — the polarity inversion applied to the IF and CASE sites only — left the check green while every conditional in every program took the wrong path, because IF declares nibble 4, CASE declares 5, and they negate into each other. The whole-suite version (M4) was caught only by luck: FOR declares C and F, whose negations D and E are declared for nothing at all. M5's first run coming back green is the entire reason H exists.

Five new non-vacuity cases, each run by hand and confirmed red before being written down:

M1  EmJcc back to `0F 8x rel16`                       -> red, 45 problems
M2  EmSetcc back to `0F 9x` + MOV AH,0                -> red, 49 problems
M3  the `>` / `>=` SETcc arms swapped (the old bug 29) -> red, 2 problems
M4  branch polarity inverted everywhere (bug 33)      -> red, 22 problems
M5  branch polarity inverted for IF and CASE only      -> red, 12 problems

Two harness faults were found while building them, both of which had produced false results before:

• The 8086 group's first draft rebuilt with make, which does not build comtest, so check_8086.py linked the stale compiler and M1 came back VACUOUS. A helper that exits non-zero makes if mutate_x; then false, so the case is silently skipped — which looks exactly like a pass. The helpers now check the rebuild and say so.
• A first cross-check of the size re-baseline reported "no mismatch" because no .COM files existed where it looked, so its loop never ran and a vacuous "all clear" came back. It was re-run with an assertion on the image count, and this time reported the real answer. Worth recording because the failure looked exactly like success — but note that cross-check was a throwaway command, not a committed script, so the guard does not guard anything permanent. run_compile_tests.sh re-baselining is still a manual edit.

Scope decisions and findings

• t33_cmpops is a new fixture, and it exists partly for clause G: =, <> and <= had never been used as a comparison anywhere in the suite, and > / >= had already been swapped once, so the coverage hole and the bug it let through were the same hole. Its .out is hand-derived from the Pascal and was written before the machine ran.
• expected.tsv re-baselined for nine control-flow fixtures, every one upward by exactly +1 per lowered site (a branch 4 -> 5 bytes, a value 5 -> 6). The site counts were counted out of the linked images and each row written only after its delta matched its own count; the arithmetic is spelled out in the file. Data sizes did not move. Byte-level expectations for those sites now live in check_8086.py, behaviour in run_com_exec.py; the matrix row only says how big the code is.
• check_8086.py cannot assert on the CASE arm's comparison: it is 3D lo hi with a non-zero label immediate, and no sound anchor can find it. The arm is covered by shape and by t22_case's execution, but the label immediate itself is unchecked.
• The check executes nothing. It proves the opcodes are 8086 and that conditions are attached to the right constructs; the control flow those bytes produce is still only checked by qemu on a 486.

Tag placement

v-TP3-8086-LOWERING points at the commit that adds this file, so its tree contains this milestone's account of it — the convention every other tag follows, and the one v-TP3-COM-IMAGE departs from for reasons recorded in its own tag message. (The tag was first placed on the work commit and moved once this file was written; both commits are in the tree, and nothing is pushed.)

What remains (next steps)

1. CmdRun (R key): in-process 8086 interpreter, cross-validated against qemu on the same images. Also the only candidate for a real 8086 execution oracle, since qemu cannot be one.
2. String variables (s : string, assignment, writeln(s)).
3. Nested procedures/recursion, var parameters (SEG:OFF), range/index checks, typed constants, array at use (t14), case subrange labels.
4. readln of a BYTE (separate xrdbyte / TU_RdByte).
5. Enforce the 4 KiB code window (error if pc reaches data start).
6. Harmonise multi-name declarations (both , and ; spellings).
7. Harden program-header parameter loop against non-advancing input.
8. FreeDOS as a third opinion (untried).