Shell: C now writes a real .COM, and one command runs every check

CmdCompile finally honours the Destination selector that TP3's Options
submenu has always been displaying.  Until now codeDest was set, shown, and
then ignored - choosing COM did nothing at all, which was a plain fidelity
gap.  Now:

  Memory  code left in the buffer, for the debugger to run in place
  .COM    WriteCom, reporting the name, the file size and the image size
  .CHN    refused, with a message

.CHN is refused rather than pretended at: it is Turbo Pascal's TBIOS overlay
format, written by the original's own overlay loader, and a silent success
would leave the user's next R with nothing to run.

ComName derives FOO.PAS -> FOO.COM in the SOURCE's directory, not the shell's
cwd, because that is what the original does - it hands the destination to the
overlay loader with the extension swapped.  It scans for the LAST dot, not the
first, because a path may contain a dotted directory component and truncating
at the first one would write the .COM somewhere else entirely.

The Destination line is now printed on every branch, not just Memory and .CHN.
It reads better and it means one thing to grep for.

tests/comtest.py closes the loop from keystroke to artifact over a pty:

    W (load) -> O (options) -> C (dest=COM) -> Q (leave) -> C (compile)
    -> the .COM is then checked ON DISK, independently: the runtime's first
       bytes, hdrFlag/hdrCS/hdrDS/hdrHeap, a fully zero code/data gap, a file
       long enough to contain the globals, and the .COM being next to the
       source rather than in the shell's cwd.

The last one is the check I would have written last and it is the one that
would have caught a cwd regression: the file is a different object than the
message that claims to have written it.

Non-vacuous: pointed at t14_types.pas, which raises ENoLib, the test goes
red on seven checks.  Noted in the docstring so a future reader does not take
it for a shell bug.

One thing the test taught me, because I had it backwards: the Options submenu
is left with Q, not ESC.  ESC is swallowed by its REPEAT loop.  The first
version of the test sent ESC, so the 'c' meant for the compiler was eaten as
a Destination change and the shell never compiled anything - the test failed
on a detail of the UI, not on the linker.

tests/run_all.sh runs everything and exits non-zero on any failure:

    build, runtime probe, compile matrix (27 asserted), .COM linker
    (24 independently checked), UI error path, UI success path

Verified from a wiped tree (every .o, binary and .lst deleted, sources kept),
because a suite that only works with warm build products is a suite that will
fail for a colleague on a fresh clone.

Also fixed the Makefile, which had accumulated a duplicated Linker.o in the
dependency list and a phase-1 link line that had lost Linker.o entirely - a
patch that had half-applied.  The module list and link line are now derived
from single variables, so they cannot disagree again.

.shell/.gitignore for the build products and for .COM files.

Verified: tests/run_all.sh OVERALL: ALL PASS, exit 0; make clean && make
rc=0, tpshell 163040 bytes.

Still not executable.  CmdRun is a stub and no .COM has been run on anything.

shell/.gitignore

@@ -0,0 +1,14 @@
+# build products
+*.o
+shell/tpshell
+shell/tpshell.lst
+shell/compiletest
+shell/comtest
+shell/modules.lst
+shell/tests/ct.lst
+__pycache__/
+
+# linker output - a .COM is a build artifact of the current source, not a
+# source file.  tests/run_com_tests.sh and tests/comtest.py create these next
+# to the fixture and remove them again, but a hand-run ComTest will not.
+*.COM

shell/Makefile

@@ -2,13 +2,20 @@ GM2   = /home/eric/bin/Modula2/Gm2/bin/gm2
 CC    = cc
 FLAGS = -fiso
 
+# The whole-program link must be two phases.  A single invocation trips gm2's
+# pass-3 rollup on a compiler-size program ("too many errors in pass 3"), so
+# the import closure is generated first and the link then consumes it.  Phase 1
+# exiting 1 is the expected rollup, hence the `|| true`.
+MODS  = Shell Term Posix TextBuf Editor Compiler Runtime Linker
+LINK  = Shell.mod Term.o Posix.o TextBuf.o Editor.o Compiler.o Runtime.o Linker.o
+
 all: tpshell
 
-tpshell: Editor.mod Shell.mod Shell.o Term.o Posix.o TextBuf.o Editor.o Compiler.o Runtime.o
-	$(GM2) $(FLAGS) -fgen-module-list=tpshell.lst -o /dev/null Shell.mod Term.o Posix.o TextBuf.o Editor.o Compiler.o Runtime.o || true
-	$(GM2) $(FLAGS) -fuse-list=tpshell.lst -o $@ Shell.mod Term.o Posix.o TextBuf.o Editor.o Compiler.o Runtime.o
+tpshell: $(MODS:%=%.o) Shell.mod Makefile
+	$(GM2) $(FLAGS) -fgen-module-list=tpshell.lst -o /dev/null $(LINK) || true
+	$(GM2) $(FLAGS) -fuse-list=tpshell.lst -o $@ $(LINK)
 
-Shell.o: Shell.mod Term.def Posix.def Editor.def TextBuf.def
+Shell.o: Shell.mod Term.def Posix.def Editor.def TextBuf.def Compiler.def Linker.def
 	$(GM2) $(FLAGS) -c Shell.mod
 
 Term.o: Term.mod Term.def Posix.def
@@ -20,7 +27,7 @@ TextBuf.o: TextBuf.mod TextBuf.def
 Editor.o: Editor.mod Editor.def Term.def TextBuf.def Posix.def
 	$(GM2) $(FLAGS) -c Editor.mod
 
-Posix.o: Posix.c
+Posix.o: Posix.c Posix.def
 	$(CC) -c Posix.c
 
 Compiler.o: Compiler.mod Compiler.def TextBuf.def Runtime.def
@@ -29,6 +36,9 @@ Compiler.o: Compiler.mod Compiler.def TextBuf.def Runtime.def
 Runtime.o: Runtime.mod Runtime.def
 	$(GM2) $(FLAGS) -c Runtime.mod
 
+Linker.o: Linker.mod Linker.def Compiler.def Posix.def
+	$(GM2) $(FLAGS) -c Linker.mod
+
 clean:
 	rm -f *.o tpshell tpshell.lst
-.PHONY: all clean
+.PHONY: all clean

shell/Shell.mod

@@ -15,7 +15,8 @@ FROM Posix IMPORT
    getcwd, chdir, opendir, readdir, closedir, statvfs,
    Dir, dirent, statvfsbuf ;
 
-FROM Compiler IMPORT Compile, CodeBytes, DataBytes ;
+FROM Compiler IMPORT Compile, CodeBytes, DataBytes, ImageBytes ;
+FROM Linker IMPORT LinkSize, WriteCom ;
 FROM Editor IMPORT Run, GotoOffset ;
 
 FROM TextBuf IMPORT
@@ -248,6 +249,45 @@ BEGIN
    END
 END SetWorkName ;
 
+(* The .COM name for the current work file: the same path with .COM.
+   TP3 derives the output name the same way (it hands the destination to the
+   overlay loader with the extension swapped), so FOO.PAS links to FOO.COM
+   in whatever directory the source lives, not to the current directory.
+
+   The dot is found by scanning for the LAST one, not the first: a Pascal
+   path may well contain a directory component with a dot in it, and taking
+   the first would truncate the directory and write the .COM somewhere else. *)
+PROCEDURE ComName (VAR dst : ARRAY OF CHAR) ;
+VAR i, dot : CARDINAL ;
+BEGIN
+   IF StrLen (workName) = 0 THEN
+      StrClear (dst) ;
+      RETURN
+   END ;
+   StrCopy (dst, workName) ;
+   dot := 0 ;
+   i := 0 ;
+   WHILE (i <= HIGH (workName)) AND (workName [i] # 0C) DO
+      IF workName [i] = "." THEN
+         dot := i
+      END ;
+      INC (i)
+   END ;
+   IF dot = 0 THEN
+      (* no extension at all - just append *)
+      StrCopy (dst, workName) ;
+      StrAppend (dst, ".COM")
+   ELSE
+      (* cut at the dot, then append *)
+      i := dot ;
+      WHILE (i <= HIGH (dst)) DO
+         dst [i] := 0C ;
+         INC (i)
+      END ;
+      StrAppend (dst, ".COM")
+   END
+END ComName ;
+
 (* ------------------------------------------------------------------ *)
 (*  buffered save of the work file: .BAK = old version, then ^Z       *)
 (* ------------------------------------------------------------------ *)
@@ -625,6 +665,7 @@ BEGIN
 END CmdMainFile ;
 
 PROCEDURE CmdCompile ;
+VAR comPath : ARRAY [0..255] OF CHAR ;
 BEGIN
    ClrScr ;
    GotoXY (1, 1) ;
@@ -633,8 +674,42 @@ BEGIN
       PutCard (CodeBytes ()) ;
       PutStr (" bytes, data ") ;
       PutCard (DataBytes ()) ;
-      PutStr (" bytes (TP3 option O not yet run)") ;
       CrLf ;
+      (* Destination, as TP3's Options submenu sets it: 0 = Memory (the
+         debugger runs the code in place), 1 = .COM, 2 = .CHN.
+         Until now the choice was displayed and then ignored, which was a
+         fidelity gap: selecting COM did nothing at all. *)
+      CASE codeDest OF
+      | 0 :
+            PutStr ("Destination: Memory (code left in the buffer)") ;
+            CrLf
+      | 2 :
+            (* .CHN is Turbo Pascal's TBIOS overlay format; the original
+               wrote it with its own overlay loader.  Refuse rather than
+               pretend - the user's next R would not find anything. *)
+            PutStr ("Destination: .CHN is not implemented") ;
+            CrLf
+      ELSE
+            PutStr ("Destination: .COM") ;
+            CrLf ;
+            ComName (comPath) ;
+            IF WriteCom (comPath) THEN
+               PutStr ("  wrote ") ;
+               PutStr (comPath) ;
+               PutStr ("  ") ;
+               PutCard (LinkSize ()) ;
+               PutStr (" bytes (image ") ;
+               PutCard (ImageBytes ()) ;
+               PutStr (")")
+            ELSE
+               (* WriteCom unlinked any partial file, so there is no half
+                  file to clean up here. *)
+               PutStr ("  could not write ") ;
+               PutStr (comPath) ;
+               PutStr (" - disk error?")
+            END ;
+            CrLf
+      END ;
       PutStr ("press ESC to return to the editor") ;
       CrLf ;
       WaitEsc

shell/build_tpshell.sh

@@ -10,22 +10,22 @@ cd "$D" || exit 9
 FLAGS="-fiso"
 
 echo "== compiling each module (isolated -c) =="
-for m in Shell Term Posix TextBuf Editor Compiler Runtime ; do
+for m in Shell Term Posix TextBuf Editor Compiler Runtime Linker ; do
    $GM2 $FLAGS -c $m.mod >/tmp/tp_c_$m 2>&1 \
       || { echo "COMPILE_FAIL $m"; grep -m3 "error:" /tmp/tp_c_$m; exit 1; }
 done
-echo "ok: Shell Term Posix TextBuf Editor Compiler Runtime"
+echo "ok: Shell Term Posix TextBuf Editor Compiler Runtime Linker"
 
 echo "== Phase 1: generate module list =="
 rm -f modules.lst
 $GM2 $FLAGS -fgen-module-list=modules.lst -o /dev/null \
-    Shell.mod Term.o Posix.o TextBuf.o Editor.o Compiler.o Runtime.o >/tmp/tp_p1 2>&1
+    Shell.mod Term.o Posix.o TextBuf.o Editor.o Compiler.o Runtime.o Linker.o >/tmp/tp_p1 2>&1
 echo "p1_rc=$?  list: $(tr '\n' ' ' < modules.lst)"
 
 echo "== Phase 2: link with the list =="
 rm -f tpshell
 $GM2 $FLAGS -fuse-list=modules.lst -o tpshell \
-    Shell.mod Term.o Posix.o TextBuf.o Editor.o Compiler.o Runtime.o >/tmp/tp_p2 2>&1
+    Shell.mod Term.o Posix.o TextBuf.o Editor.o Compiler.o Runtime.o Linker.o >/tmp/tp_p2 2>&1
 echo "p2_rc=$?"
 grep -cE "error:|undefined" /tmp/tp_p2
 ls -l tpshell 2>/dev/null | awk '{print "tpshell bytes:",$5}'

shell/modules.lst

@@ -1,3 +0,0 @@
-TextBuf
-SYSTEM
-Compiler

shell/tests/comtest.py

@@ -0,0 +1,136 @@
+#!/usr/bin/env python3
+"""End-to-end UI test: C writes a real .COM, verified by reading the disk.
+
+uitest.py covers the error path (compile fails -> editor opens on the error).
+This covers the success path, which is the one that produces an artifact:
+
+  1. W loads a work file (LoadWorkFile ends with a Pause: needs a filler key)
+  2. O opens the Options submenu and C selects Destination = .COM
+  3. ESC returns to the main menu
+  4. C compiles, and the shell reports the .COM it wrote
+  5. The .COM is then INDEPENDENTLY checked on disk: the runtime's first
+     bytes, a fully zero code/data gap, a size that covers the data area, and
+     the program header words
+
+Step 5 is the point.  "The shell said it wrote a file" is not evidence that
+the file is right, and the shell reports success from the same state that
+produced the bytes, so the check that matters happens outside the shell.
+
+The .COM is removed afterwards so the test can be re-run.
+
+Usage: comtest.py [fixture.pas]
+
+Passing a fixture that does NOT compile is expected to be RED - that is the
+test being non-vacuous, not a failure of the shell.  It was checked that way
+(t14_types.pas, which raises ENoLib for 'array' at its point of use).
+"""
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+from ptyharness import SHELL_DIR, Screen, drain, reap, send, spawn, status_str, visible
+
+FIXTURE = os.path.abspath(sys.argv[1]) if len(sys.argv) > 1 else os.path.join(
+    os.path.dirname(os.path.abspath(__file__)), "fixtures", "t26_str_mixed_args.pas")
+COM = os.path.splitext(FIXTURE)[0] + ".COM"
+
+# Restated here on purpose - the checker must not ask the code under test.
+RTSZ = 385                    # Runtime.RT_Size()
+DATAB = RTSZ + 0x1000         # Compiler: data base = rtSz + 1000H
+HEAD = "8B C0 8B 54 04 8B 4C 08"   # initmem: MOV AX,AX / MOV DX,[SI+4] / MOV CX,[SI+8]
+
+
+def check_com(path, expected_src_len):
+    """Return a list of problems (empty = the .COM is correct)."""
+    errs = []
+    if not os.path.exists(path):
+        return ["no .COM file was written"]
+    d = open(path, "rb").read()
+    if d[:8].hex(" ").upper() != HEAD:
+        errs.append("runtime not at offset 0: first bytes %s" % d[:8].hex(" ").upper())
+    if len(d) < DATAB:
+        errs.append("file is %d bytes, shorter than the data base %d - the "
+                    "globals would be outside the file" % (len(d), DATAB))
+    # the runtime must be followed immediately by the program header,
+    # whose first word is the "image is valid" flag
+    if len(d) >= RTSZ + 2:
+        flag = int.from_bytes(d[RTSZ:RTSZ + 2], "little")
+        if flag != 1:
+            errs.append("hdrFlag=%d at the program header, want 1" % flag)
+    if len(d) >= RTSZ + 10:
+        ds = int.from_bytes(d[RTSZ + 4:RTSZ + 6], "little")
+        heap = int.from_bytes(d[RTSZ + 6:RTSZ + 8], "little")
+        if ds != DATAB:
+            errs.append("hdrDS=%d, want %d" % (ds, DATAB))
+        if heap < ds:
+            errs.append("hdrHeap=%d < hdrDS=%d" % (heap, ds))
+    # the gap between the end of the code and the data area must be all zero
+    cs = int.from_bytes(d[RTSZ + 2:RTSZ + 4], "little") if len(d) >= RTSZ + 4 else 0
+    if cs and cs < DATAB:
+        nz = sum(1 for b in d[cs:DATAB] if b)
+        if nz:
+            errs.append("%d non-zero bytes in the code/data gap" % nz)
+    return errs
+
+
+def main():
+    if os.path.exists(COM):
+        os.unlink(COM)
+    pid, fd = spawn()
+    checks = []
+    try:
+        drain(fd, quiet=0.6)
+        send(fd, b"w")
+        send(fd, FIXTURE.encode())
+        send(fd, b"\r", quiet=0.8)
+        send(fd, b" ", quiet=0.5)              # clears LoadWorkFile's Pause
+
+        # Options submenu -> Destination = .COM.
+        # The submenu is left with Q (that is its Quit key, as in the
+        # original's optmenu); ESC is swallowed by its REPEAT loop.
+        send(fd, b"o", quiet=0.8)
+        opt = visible(send(fd, b"c", quiet=0.5))
+        checks.append(("Options submenu opened", "Compile" in opt))
+        checks.append(("selecting C set Destination = COM", "COM" in opt))
+        after = visible(send(fd, b"q", quiet=0.8))
+        checks.append(("Q left the submenu for the main menu", "Main file" in after))
+
+        comp = visible(send(fd, b"c", quiet=2.0))
+        checks.append(("compile reported success", "Compiled OK" in comp))
+        checks.append(("shell reported the Destination it used",
+                       "Destination" in comp))
+        checks.append(("shell named the .COM it wrote",
+                       os.path.basename(COM) in comp))
+
+        send(fd, b"\x1b", quiet=0.6)
+        send(fd, b"q", quiet=0.8)
+    finally:
+        status = reap(fd, pid)
+
+    checks.append(("shell exited cleanly (status 0)", os.WIFEXITED(status)
+                   and os.WEXITSTATUS(status) == 0))
+
+    # now verify the artifact independently
+    errs = check_com(COM, len(open(FIXTURE, "rb").read()))
+    checks.append((".COM exists on disk and passes the independent check"
+                   if not errs else ".COM check: " + "; ".join(errs), not errs))
+    checks.append((".COM was written next to the source, not in the shell's cwd",
+                   os.path.dirname(os.path.abspath(COM)) == os.path.dirname(FIXTURE)))
+
+    print("UI TEST: %s" % os.path.basename(FIXTURE))
+    print("-" * 60)
+    ok = True
+    for name, passed in checks:
+        print("%-58s %s" % (name[:58], "PASS" if passed else "FAIL"))
+        ok = ok and passed
+    print("-" * 60)
+    print("child: %s" % status_str(status))
+    print("RESULT: %s" % ("ALL PASS" if ok else "FAILURES PRESENT"))
+    if os.path.exists(COM):
+        print("(removed %s)" % COM)
+        os.unlink(COM)
+    return 0 if ok else 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

shell/tests/run_all.sh

@@ -0,0 +1,75 @@
+#!/bin/bash
+# Run every check.  Non-zero exit if anything fails.
+#
+# Four suites, in increasing order of "how much could be lying to me":
+#
+#   1. compile matrix   27 fixtures, verdict + code size + data size ASSERTED
+#                       from expected.tsv.  Fast, no pty.
+#   2. .COM linker      links every fixture, then re-verifies the bytes with an
+#                       independent checker that restates the layout constants
+#                       instead of asking the compiler.
+#   3. UI error path    pty: W, C on a bad file, editor opens on the error,
+#                       Ctrl-K D, Q.  Covers the shell behaviour that only
+#                       exists interactively.
+#   4. UI success path  pty: Options -> Destination=COM, C, then the .COM is
+#                       checked ON DISK.  This is the end-to-end path from
+#                       keystroke to artifact.
+#
+#   RtProbe             dumps the runtime size and its 14 entry offsets, so a
+#                       runtime change that moves an entry is visible here.
+#
+# rt_exec.py is NOT run: it needs Unicorn, whose 16-bit ModRM decoding is
+# wrong on this machine (see SUMMARY.md), so its failures would be the
+# emulator's, not the runtime's.  Running it by default would be noise.
+set -u
+D=/home/eric/Projets/Projets-Modula2/MyWork/TP3-comp/shell
+GM2=/home/eric/bin/Modula2/Gm2/bin/gm2
+cd "$D" || exit 9
+
+fail=0
+run () {
+   name=$1
+   shift
+   echo
+   echo "=============================================================="
+   echo "== $name"
+   echo "=============================================================="
+   if "$@" ; then
+      echo "-- $name: PASS"
+   else
+      echo "-- $name: FAIL (rc=$?)"
+      fail=1
+   fi
+}
+
+echo "== build =="
+make clean >/dev/null 2>&1
+if ! make >/tmp/tp_all_mk 2>&1 ; then
+   echo "BUILD FAIL"
+   grep -m10 "error:" /tmp/tp_all_mk
+   exit 1
+fi
+echo "make rc=0, tpshell $(stat -c%s tpshell) bytes"
+# gm2 reports a pass-3 rollup on the compiler during phase 1 of the two-phase
+# link; that is expected and is not a build failure (make rc=0 is the check).
+
+run "runtime probe"  bash -c '
+   GM2=/home/eric/bin/Modula2/Gm2/bin/gm2
+   $GM2 -fiso -o /tmp/tp_rtprobe tests/RtProbe.mod Runtime.o Posix.o \
+       >/tmp/tp_rt 2>&1 || { grep -m5 error: /tmp/tp_rt; exit 1; }
+   /tmp/tp_rtprobe'
+
+run "compile matrix"  tests/run_compile_tests.sh
+run "COM linker"      tests/run_com_tests.sh
+run "UI error path"   python3 tests/uitest.py
+run "UI success path" python3 tests/comtest.py
+
+echo
+echo "=============================================================="
+if [ "$fail" -eq 0 ]; then
+   echo "OVERALL: ALL PASS"
+else
+   echo "OVERALL: FAILURES PRESENT"
+fi
+echo "=============================================================="
+exit $fail