Prechádzať zdrojové kódy

feat(compiler): GENZ80 recon + MCD-diff harness

GENZ80.md: overlay interface/trigger/recovery (binary absent
everywhere). tools/mcdiff.sh: stage-1 static checks green + stage-2
VM procedure; VM builds. Harness already fixed 5 leftover
renames.
Eric Streit 1 týždeň pred
rodič
commit
836b11a12c

+ 10 - 0
SESSION.md

@@ -26,6 +26,16 @@ proc32 WriteOutputFiles added; 8 SYMTAB + 9 EXPRESS signature swaps applied
 all KEEP; D1/D2 resolved (map stands); CappedStrLen=StrLen alias noted.
 all KEEP; D1/D2 resolved (map stands); CappedStrLen=StrLen alias noted.
 Next: GENZ80 recon + MCD-diff verification harness.
 Next: GENZ80 recon + MCD-diff verification harness.
 
 
+## 2026-10-03: GENZ80 recon + MCD-diff harness (`genz80-harness-01`)
+GENZ80: no binary/disassembly anywhere (only call site + overlay-table
+strings); trigger chain partly open (no setter of nativeCodeRequested found);
+recovery = extract overlay from original CP/M media. Harness tools/mcdiff.sh:
+stage 1 green (1a clean, 1b FORWARD-OK, 1d spot-check; fixed StrLenHelper,
+word6, Doubles.proc5, Scanner.word2, Compiler.word3 leftovers on the way).
+VM builds clean (gcc, `m2` binary); stage-2 run blocked on terminal I/O
+(documented pty approach in mcdiff.sh header).
+Next: GENZ80 media hunt; stage-2 pty-driven MCD diff per module.
+
 ## Next steps (in order)
 ## Next steps (in order)
 
 
 1. DONE (`statemen-draft-v1`): `STATEMEN.MOD` 830 lines + frame-mechanics proof.
 1. DONE (`statemen-draft-v1`): `STATEMEN.MOD` 830 lines + frame-mechanics proof.

+ 1 - 1
src/compiler/CODEGEN.MOD

@@ -389,7 +389,7 @@ BEGIN
         CASE entry^.word0 OF
         CASE entry^.word0 OF
         | 0: (* 02EE *)
         | 0: (* 02EE *)
             IF entry^.word2 <> 0 THEN Emit2(2, entry^.word2)
             IF entry^.word2 <> 0 THEN Emit2(2, entry^.word2)
-            ELSE Emit2(OPCALLREL, Scanner.StrLenHelper(entry^.word1, 128));
+            ELSE Emit2(OPCALLREL, Scanner.StrLen(entry^.word1, 128) (*FIXME: open-array HIGH passed explicitly*));
                  EmitString(entry^.word1);
                  EmitString(entry^.word1);
             END;
             END;
         | 1: (* 0308 *)
         | 1: (* 0308 *)

+ 3 - 3
src/compiler/EXPRESS.MOD

@@ -22,11 +22,11 @@
    order-correct. Applied: MatchOpClass(allowed, errCode) [was swapped],
    order-correct. Applied: MatchOpClass(allowed, errCode) [was swapped],
    CentralError(ok, code) [was 1-param]. Full per-proc signature audit PENDING.
    CentralError(ok, code) [was 1-param]. Full per-proc signature audit PENDING.
    Numbered-module mappings applied (verified against .DEF orders):
    Numbered-module mappings applied (verified against .DEF orders):
-   - Compiler.word6..26 = IntType..compilationActive (DEF order = word order).
-   - Scanner.word2/5/6/7/8/9/10/11/12 = scanOptions/curSymbol/isLiteral/
+   - Compiler.IntType..26 = IntType..compilationActive (DEF order = word order).
+   - Scanner.scanOptions/5/6/7/8/9/10/11/12 = scanOptions/curSymbol/isLiteral/
      cardValue/curNode/identKind/followSet/tokenBuffer/literalType.
      cardValue/curNode/identKind/followSet/tokenBuffer/literalType.
      Scanner.word13: unknown, kept + FIXME.
      Scanner.word13: unknown, kept + FIXME.
-   - CodeGen.word5 = emitEnabled. Doubles.proc5/6/7/8/9/11 =
+   - CodeGen.word5 = emitEnabled. Doubles.qcp/6/7/8/9/11 =
      qcp/qadd/qsub/qmul/qdiv/qneg (DEF order = proc order).
      qcp/qadd/qsub/qmul/qdiv/qneg (DEF order = proc order).
    - Single-arg BaseTypeOf(X) calls = TypeKindOf (proc27); two-arg keeps
    - Single-arg BaseTypeOf(X) calls = TypeKindOf (proc27); two-arg keeps
      BaseTypeOf (proc34). ParseCondValue() = BoolCondHelper (proc12).
      BaseTypeOf (proc34). ParseCondValue() = BoolCondHelper (proc12).

+ 75 - 0
src/compiler/GENZ80.md

@@ -0,0 +1,75 @@
+# GENZ80 recon: the missing native backend
+
+## What it is
+
+`GENZ80` is a dynamically loaded overlay (`Loader.Call("GENZ80")`), invoked
+from `Scanner.Compile` (see `SCANNER.MOD` `Compile` body) AFTER the `COMPILE`
+link finishes — iff `Compiler.nativeCodeRequested` (global25, init FALSE in
+`InitCompiler`). It reads back the just-emitted MCode window and generates
+native Z80 machine code:
+
+```modula2
+Loader.Call("COMPILE");
+IF Compiler.nativeCodeRequested THEN
+  CheckStackMark(codeBuffer + CodeGen.nextEmitPos);
+  IF CodeGen.windowBase <> 0 THEN
+    MOVE(codeBuffer, codeBuffer + CodeGen.windowBase,
+         CodeGen.nextEmitPos - CodeGen.windowBase);
+    Files.SetPos(codeFile, LONG(0));
+    IF Files.ReadBytes(codeFile, codeBuffer, CodeGen.windowBase)
+       <> CodeGen.windowBase THEN RAISE Files.EndError END;
+  END;
+  Files.SetPos(codeFile, LONG(0));
+  Loader.Call("GENZ80");
+END;
+```
+
+Failure mode is uniform with other overlays (`Loader.LoadError` →
+`ERROR: CANNOT LOAD OVERLAY`, delete partial output).
+
+## Trigger chain (partly open)
+
+* Shell run menu sets `ComLine.execute` = 4 (bytecode) vs 5 (native Z80)
+  (`shell.txt` @1351, explicit comment in disassembly).
+* `CodeGen.InitCodeGenerator` reads `execute` (`checkOverflow := execute = 4`).
+* Setter of `Compiler.nativeCodeRequested=TRUE` NOT FOUND in any recovered
+  code: no `store COMPILER.word25` exists in `compiler/scanner/symtab/pass1/
+  statement/express/codegen/shell.txt` (only the FALSE init in `compiler.txt`
+  @07ec). Candidates: shell compile path (no COMPILER dependency though),
+  `Scanner.Compile` prologue (not in the decompiled body — which may itself
+  be incomplete), or a lost `%`-option branch. OPEN QUESTION for verification.
+
+## Recovery status: binary absent
+
+No GENZ80 bytes exist in either upstream repo:
+
+* `MCode_disassembly/` has no `genz80.txt` (all other overlays present).
+* `m2_mcd.txt` (resident M2.COM image) contains KERNEL COMLINE EDITOR
+  COMPILER SYMTAB DOUBLES TERMINAL LOADER CODEGEN ERRORS EDIT2 TEXTS FILES
+  CONVERT EXEC SCANNER EDITDISK — no GENZ80, no PASS1/STATEMEN/EXPRESS.
+* `COMPILE.DIR/` has only COMPILE/PASS1/STATEMEN/EXPRESS `.MCD`.
+* Only traces: the `"GENZ80"` string in `SCANNER.MCD`'s overlay table and in
+  `SYSTEM.DIR/SYSTEM{,0,1}.OVR` next to `BADOVERLAY` / `CANNOT LOAD OVERLAY`.
+
+So there is nothing to decompile yet. Recovery plan:
+
+1. Source an original Turbo Modula-2 CP/M disk image (M2.COM + overlays).
+   The overlay lives either as a separate `GENZ80.OVL/.COM` file or appended
+   in M2.COM's overlay area — check the CP/M directory + `SYSTEM.OVR`-style
+   loader tables first.
+2. Run it through `unassemble.c` to produce `MCode_disassembly/genz80.txt`
+   (same procedure as the other overlays).
+3. Decompile proc-by-proc into `src/compiler/GENZ80.MOD` (expected shape: a
+   single-module MCode→Z80 peephole translator over `Scanner.codeBuffer`,
+   resolving the `OpenFixup`/`PokeCodeByte` back-patches natively).
+4. Reloaded note: the portable targets run MCode everywhere, so GENZ80 is
+   only needed for the "authentic CP/M + native .COM output" story — it stays
+   optional and out of the default build.
+
+## Interface contract (for the future draft)
+
+* Input: complete MCode image staged at `Scanner.codeBuffer`/`codeFile`
+  (`nextEmitPos` bytes, `windowBase` rewound by the caller above).
+* Uses: `Scanner.*` (source/options), `Compiler.*` (predefs), `CodeGen.*`
+  (opcodes/fixup helpers), `Files.*` (output).
+* Must NOT disturb `SymTab` heaps (they sit above the code window).

+ 1 - 1
src/compiler/PASS1.MOD

@@ -982,7 +982,7 @@ PROCEDURE InitPass1; FORWARD;
 (*{0db2} proc32 — WriteOutputFiles: file-output driver (.SYM via
 (*{0db2} proc32 — WriteOutputFiles: file-output driver (.SYM via
   SymTab.SaveSymbolFile vs .MCD via WriteMcdHeader+EmitTypeDesc).
   SymTab.SaveSymbolFile vs .MCD via WriteMcdHeader+EmitTypeDesc).
   0 params. Refutes FRONTEND.md ~SkipToDeclStart (only the head loop
   0 params. Refutes FRONTEND.md ~SkipToDeclStart (only the head loop
-  skips input). Compiler.word3/word2 flag roles TBD (no DEF slots). *)
+  skips input). Compiler.scopeSpare3/word2 flag roles TBD (no DEF slots). *)
 PROCEDURE WriteOutputFiles;
 PROCEDURE WriteOutputFiles;
 BEGIN
 BEGIN
   WHILE Scanner.curChar # CHR(255) DO (*{0db4-0db9} *)
   WHILE Scanner.curChar # CHR(255) DO (*{0db4-0db9} *)

+ 1 - 0
src/compiler/README.md

@@ -15,6 +15,7 @@ No `0x1A`/`0x00` padding, LF line endings.
 | `SYMTAB.DEF` | expanded (words 10–27 mapped) + `StringPtr` + draft signatures |
 | `SYMTAB.DEF` | expanded (words 10–27 mapped) + `StringPtr` + draft signatures |
 | `SYMTAB.MOD` | **DRAFT v1, unverified**: full bodies assembled from 4 group passes; see header for conflicts/FIXMEs |
 | `SYMTAB.MOD` | **DRAFT v1, unverified**: full bodies assembled from 4 group passes; see header for conflicts/FIXMEs |
 | `FRONTEND.md` | export map for `PASS1/STATEMEN/EXPRESS` (+ newly found EXPRESS internals) |
 | `FRONTEND.md` | export map for `PASS1/STATEMEN/EXPRESS` (+ newly found EXPRESS internals) |
+| `GENZ80.md` | recon: native backend binary absent (interface + trigger + recovery plan) |
 | `EXPRESS.MOD` | **DRAFT v1, unverified** (CentralError 2-param + MatchOpClass order fixed 2026-10-03) |
 | `EXPRESS.MOD` | **DRAFT v1, unverified** (CentralError 2-param + MatchOpClass order fixed 2026-10-03) |
 | `STATEMEN.MOD` | **DRAFT v1, unverified**: 830 lines; frame-mechanics proof in header |
 | `STATEMEN.MOD` | **DRAFT v1, unverified**: 830 lines; frame-mechanics proof in header |
 | `PASS1.MOD` | **DRAFT v1, unverified** (+proc32 `WriteOutputFiles`, +audit swaps) |
 | `PASS1.MOD` | **DRAFT v1, unverified** (+proc32 `WriteOutputFiles`, +audit swaps) |

+ 107 - 0
tools/mcdiff.sh

@@ -0,0 +1,107 @@
+#!/bin/bash
+# mcdiff.sh — verification harness for the src/compiler re-read.
+# Stage 1 (runs HERE, no VM needed): static consistency checks.
+# Stage 2 (needs the Reloaded VM + original compiler): true MCD diff;
+#   procedure documented below. Exit nonzero on any stage-1 failure.
+#
+# Stage 2 procedure (once, per module MOD in dependency order
+#   Compiler Scanner SymTab Errors CodeGen Express Statemen Pass1):
+#   1. Build the VM:  cd Turbo-Modula-2-Reloaded-main/vm_linux && make
+#   2. Stage sources: copy src/compiler/X.{DEF,MOD} onto m2.dsk
+#      (mtools: mcopy -i m2.dsk, or sudo mount -o loop).
+#      File names must be 8+3 uppercase (COMPILER.DEF etc. already are).
+#   3. Boot: ./m2_disk m2.dsk ; at the M2 prompt run the resident
+#      compiler on each MOD (c <file>), producing X.MCD on the image.
+#   4. Extract: mcopy -i m2.dsk ::COMPILER.DIR/X.MCD ./build/
+#   5. Disassemble both: unassemble.c built from Reversing-Turbo-Modula2-main
+#      (cc -o unassemble unassemble.c), run on original .MCD from
+#      Turbo-Modula-2-Reloaded-main/disk_content/COMPILER.DIR/ and on build/.
+#   NOTE 2026-10-03: the VM needs a real terminal (termios raw mode); plain
+#   pipes hang. Drive it under a pty, e.g.
+#     script -qec "./vm_linux/m2 m2.dsk" /dev/null
+#   or expect(1), sending keys with delays (CR-terminated lines). mtools is
+#   NOT installed in this environment — staging currently needs manual mount.
+#   6. Diff against Reversing-Turbo-Modula2-main/MCode_disassembly/<mod>.txt.
+#      Byte-identical MCD (hence identical disassembly modulo addresses) =
+#      proof of a faithful rename/decompile. Record result per module in
+#      docs/compiler/70-rename-plan.md style commit notes.
+set -u
+SRC="${1:-src/compiler}"
+fail=0
+
+say() { printf '%s\n' "$*"; }
+err() { printf 'FAIL: %s\n' "$*" >&2; fail=1; }
+
+say "== stage 1a: numbered identifiers (must be zero outside comments) =="
+# strip (* *) comments, then look for procN/globalN/paramN/localN/PROC N
+tmp=$(mktemp -d)
+for f in "$SRC"/*.MOD "$SRC"/*.DEF; do
+  [ -f "$f" ] || continue
+  python3 - "$f" "$tmp" <<'PY'
+import re, sys, pathlib
+src = pathlib.Path(sys.argv[1]).read_text()
+nos = re.sub(r'\(\*.*?\*\)', '', src, flags=re.S)  # drop (* *) comments
+hits = re.findall(r'\b(?:global|proc|param|local)\d+\b|\bPROC\d+\b', nos)
+if hits:
+    from collections import Counter
+    print(f'{sys.argv[1]}: {dict(Counter(hits))}')
+PY
+done | tee "$tmp/hits" | grep . && err "numbered identifiers in code" || say "clean"
+
+say "== stage 1b: FORWARD/def consistency per MOD =="
+python3 - "$SRC" <<'PY'
+import re, sys, pathlib
+bad = False
+for f in sorted(pathlib.Path(sys.argv[1]).glob('*.MOD')):
+    nos = re.sub(r'\(\*.*?\*\)', '', f.read_text(), flags=re.S)
+    fwds = re.findall(r'PROCEDURE\s+(\w+)\s*(?:\([^;]*\))?(?:\s*:\s*\w+)?\s*;\s*FORWARD\s*;', nos)
+    defs = re.findall(r'PROCEDURE\s+(\w+)\s*(?:\([^;]*\))?(?:\s*:\s*\w+)?\s*;', nos)
+    from collections import Counter
+    dc = Counter(defs)
+    dupes = [k for k, v in dc.items() if v > 1 and k not in fwds]
+    missing = [w for w in fwds if w not in defs]
+    if dupes or missing:
+        bad = True
+        print(f'{f.name}: dupes={dupes} missing-defs={missing}')
+print('FORWARD-OK' if not bad else 'FORWARD-BAD')
+PY
+
+say "== stage 1c: FIXME inventory =="
+grep -c "FIXME" "$SRC"/*.MOD "$SRC"/*.DEF 2>/dev/null || true
+
+say "== stage 1d: unknown cross-module callees (spot check) =="
+# every Module.Proc call should resolve to a renamed export or a flagged FIXME nearby
+python3 - "$SRC" <<'PY'
+import re, sys, pathlib
+known = set()
+for f in pathlib.Path(sys.argv[1]).glob('*.DEF'):
+    nos = re.sub(r'\(\*.*?\*\)', '', f.read_text(), flags=re.S)
+    mod = f.stem.capitalize()
+    for m in re.findall(r'PROCEDURE\s+(\w+)', nos):
+        known.add(f'{mod}.{m}')
+    for m in re.findall(r'^\s{4}(\w+)\s*[:,\[]', nos, re.M):
+        known.add(f'{mod}.{m}')
+# + known drafts without DEF
+for mod, procs in {
+  'Express': ['ParseExpression','ParseSimpleExpression','EmitRangeCheck','EmitCompare','ParseDesignatorTail','ParseTerm','ParseFactor','ParseSetCons','ParseFuncCall','ParseTypeCast','ExpandStdProc43','ExpandStdProc44','ExpandStdProc45','ExpandStdProc46','ParseSetOrCastTail','LoadOperand','StoreOperand','FoldRelation','CheckRelationTypes','GetConstVal','GetTypeDesc','ParseAssignment','IsOrdinal','IsSet','BaseTypeOf','FoldConstOp','ConstToCard','ConstToInt','MatchOpClass','CheckAssignable','ParseSelector','PushExprDesc','PopExprDesc','CentralError','EvalConstExpr'],
+  'Statemen': ['ParseStatement','GuardedStatSeq','ParseBeginBlock','ParseCaseStmt','ParseCaseTail','ParseFor','ParseWhile','ParseRepeat','ParseIf','ParseLoop','ParseExit','ParseResult','ParseWith','ParseReturn','CalcMemOffset','SyncToStatStart','ParseStatSeqLoop','EmitJumpFixup','NextSymWrap','CloseBlockChain','EmitNameHash','EmitComBytes'],
+  'Pass1': ['ParseType','ParseFormalParams','ParseValParam','ParseVarParam','ParseFieldList','EvalConstExpr','CheckConformant','ParseRecordType','CloseScopeFixup','ParseConstDecl','ParseVarDecl','GetIdent','ParseImportItem','OpenScopeEmit','TypeSizeRec','ParseProcedureDecl','ParseModuleHead','ParseExportList','ParseOneDecl','ParseDeclarations','ParseImportFrom','ParseModuleBlock','ParseProcBody','ParseMainModule','WriteMcdHeader','WriteOutputFiles','EmitConstVal','EmitTypeDesc','PushTempMark','PopTempMark','AllocSymEntry','LinkSymEntry','InsertSetConst','CondInsert','InsertEntry','CheckForwardResolved','Pass1Main','InitPass1','EmitModuleProlog','ParseDeclSync','ParseConstOrTypeArm','ParseModuleArm','TryParseDecl','ParseImportScope','CappedStrLen'],
+  'SymTab': ['FindModule','EnterModule','LookupChain','MatchSym','SymbolAssert','CheckNode','NewString','GetName','PutAttr','GetAttr','AllocSym','EnterChained','EnterIdent','UnlinkScope','InitHeaps','InsertBuiltin','EnterStandards','LoadSymFile','FindExport','ListExports','ImportOne','ImportAll','SkipQualident','DefineDefinitionModule','BuildBitsetConst','DefineProgramModule','InitSymbolTables','NoteImported','DoImportList','CompileBody','CheckForwards','TryCompile','CompileDispatch','Checksum7w','SaveSymbolFile','RestoreAfterEdit'],
+}.items():
+    known.update(f'{mod}.{p}' for p in procs)
+import collections
+susp = collections.Counter()
+for f in sorted(pathlib.Path(sys.argv[1]).glob('*.MOD')):
+    for i, line in enumerate(f.read_text().splitlines(), 1):
+        code = line.split('(*')[0]
+        for m in re.findall(r'\b(Scanner|Errors|CodeGen|Compiler|SymTab|Express|Statemen|Pass1|Loader|Files|Texts|Terminal|ComLine|Doubles|Editor|Storage|System)\.(\w+)', code):
+            q = f'{m[0]}.{m[1]}'
+            if q not in known and m[1] not in ('WriteString','WriteLn','WriteChar','WriteCard','SetCol','ReadBytes','WriteBytes','SetPos','Open','Close','Create','Delete','Allocate','MARK','RELEASE','MOVE','ADR','BITSET','CARDINAL','ADDRESS','ORD','CHR','CAP','HIGH','NIL','TRUE','TRUE','HALT','RETURN','INC','DEC','INCL','EXCL','IN','ABS','ODD','SIZE','HALT','Close','Create','Delete','Open','ReadBytes','WriteBytes','SetPos','FileSize','NextPos','NoTrailer','ReadByte','WriteByte','ResetSys','UseError','EndError','DiskFull','Call','LibOpen','LibRead','InitLibs','LoadError','firstDrive','Edit','editPosition','editOkFlag','ReadChar','ReadAgain','numCols','commandLine','inName','outName','codepos','execute','RedirectOutput','PromptFor','qcp','qadd','qsub','qmul','qdiv','qneg','qabs','StrToDouble','DerefAliasType','StrLen','FindIdent','InsertSymbol','CopyStringToHeap','Allocate','GetStackMark','CheckStackMark','OpenScope','NewNode','NewSizedNode','DeclareIdentifier','MakeIdentifierNode','ExpectStringLiteral','ChangeFileExtension','GetSym','AcceptSymbol','ExpectSymbol','TestSymbolInSet','NeedIdentifier','ReportError'):
+                susp[(f.name, q)] += 1
+for (f, q), n in sorted(susp.items()):
+    print(f'{f}:{q} x{n}')
+PY
+
+if [ "$fail" -ne 0 ]; then say "STAGE 1 FAILED"; exit 1; fi
+say "STAGE 1 PASSED"
+rm -rf "$tmp"