docs: analyse communications and ModBase toolkits

docs/INDEX.md

@@ -36,8 +36,9 @@ bisecting *why* a decision was made, not for reading end-to-end).
 
 - **`future-work-2026-10-25.md`** — TBITSIZE, ADDRESS arithmetic, set
   ordering, complex types, pragmas (TopSpeed vs portable), UTF-8 string
-  libraries, collections and the TopSpeed B-tree Toolkit, PIM vs ISO
-  coroutines, external assembly, dynamic-set design and suggested order.
+  libraries, collections and the TopSpeed B-tree Toolkit, communications,
+  pyserial, ModBase, PIM vs ISO coroutines, external assembly,
+  dynamic-set design and suggested order.
 
 ## Designs / running records
 

docs/future-work-2026-10-25.md

@@ -3,9 +3,9 @@
 This note records the design discussions from 2026-10-25 so the
 questions do not have to be re-derived later.  It covers `TBITSIZE`,
 address arithmetic, set ordering, complex types, pragmas, UTF-8 string
-libraries, collections, the TopSpeed B-tree Toolkit, PIM versus ISO
-coroutines, external assembly, and the dynamic-set design.  It is not a
-commitment to implement everything.
+libraries, collections, the TopSpeed B-tree Toolkit, communications,
+pyserial, ModBase, PIM versus ISO coroutines, external assembly, and the
+dynamic-set design.  It is not a commitment to implement everything.
 
 **State at the time of writing:** suite 310/310, one-shot fixpoint OK
 (3,845,439 bytes), per-object fixpoint OK (11 objects), corpus
@@ -1046,7 +1046,295 @@ The TopSpeed B-tree Toolkit should not delay steps 1–3.
 
 ---
 
-## 11. Suggested order of future work
+## 11. Communications and ModBase toolkits
+
+These are three different things and should not be merged into one
+design:
+
+- the TopSpeed Communication Toolkit: DOS/OS-2 serial driver, file
+  transfer protocols, terminal emulation and scripting;
+- pyserial: a modern portable serial transport library (transport only);
+- ModBase: a dBase III-compatible database engine.
+
+All TopSpeed/ModBase source is commercial historical code; use it as a
+design reference, not copyable source.  pyserial is BSD-3-Clause.
+
+### 11.1 Current V3 state
+
+V3 has no serial, socket, terminal-emulation, scripting or database
+library.  What exists:
+
+- `lib/iso/IOChan`, `StreamFile`, `Files` for byte/line channels;
+- `lib/common/SysIO` and `runtime/syslib/SysShim` for stdio/file I/O;
+- `EXTERNAL` and the C shim for libc access;
+- PIM coroutines (no general scheduler);
+- `Storage`/`SysStorage`.
+
+### 11.2 TopSpeed Communication Toolkit
+
+References:
+
+```text
+$HOME/bin/Dos/M2/TS-V3/SRC/RS232.MOD
+$HOME/bin/Dos/M2/TS-V3/SRC/XMODEM.MOD
+$HOME/bin/Dos/M2/TS-V3/SRC/YMODEM.MOD
+$HOME/bin/Dos/M2/TS-V3/SRC/WXMODEM.MOD
+$HOME/bin/Dos/M2/TS-V3/SRC/KERMIT.MOD
+$HOME/bin/Dos/M2/TS-V3/SRC/TERM.MOD
+$HOME/bin/Dos/M2/TS-V3/SRC/SCRIPTCO.MOD
+$HOME/bin/Dos/M2/TS-V3/SRC/SCRIPTPR.MOD
+$HOME/bin/Dos/M2/TS-V3/INCLUDE/RS232.DEF
+$HOME/bin/Dos/M2/TS-V3/INCLUDE/FTSupport.DEF
+$HOME/bin/Dos/M2/TS-V3/Manuals/Modula2-Communication-toolkit.odt
+```
+
+The low-level `RS232` driver:
+
+- COM1–COM4 with fixed I/O addresses;
+- direct 8250/16550 UART register access;
+- interrupt vectors and interrupt-controller programming;
+- 1024-byte TX and RX ring buffers;
+- RTS/CTS or XON/XOFF flow control;
+- high/low water marks at 3/4 and 1/4 of the RX buffer;
+- `SerialRead` timeout in tenths of a second;
+- `SerialWrite` copies into the TX ring;
+- DTR/RTS/DCD control and break generation;
+- TX/RX hooks for monitoring;
+- separate DOS and OS/2 implementations.
+
+Limitations:
+
+- singleton driver, not a multi-port object model;
+- `SelectPort` disables the previous handler;
+- `InitPort`/`Install` have no error result;
+- busy-waiting on full TX buffer and read timeout;
+- no read/write cancellation;
+- no buffer-level query;
+- no clean open/close lifecycle (only `Install`/`UnInstall`);
+- fixed port table and hardware assumptions.
+
+The protocol layer:
+
+- XMODEM (checksum/CRC negotiation);
+- YMODEM, YMODEM Batch, YMODEM-g;
+- Windowed XMODEM;
+- Kermit;
+- shared `FTSupport` global state and `FBuff`;
+- CRC table and status/progress variables.
+
+The protocol implementations are useful as protocol state-machine
+references, but they call `RS232` and `FIO` directly and drive status
+windows through `CommUtil`/`Window`.  They are not transport-agnostic.
+Only one transfer can be active at a time.
+
+The toolkit also includes:
+
+- VT100, VT52 and ANSI terminal emulation;
+- keyboard driver;
+- menu library;
+- ProComm-like compiled script language;
+- scanner, compiler, pseudo-code interpreter;
+- cooperative `SingleStep` execution;
+- `WAITFOR`, `WHEN`, macros, logging, file-transfer commands.
+
+The script VM is a useful design for an application-level automation
+engine, not a general V3 library requirement.
+
+### 11.3 pyserial as the transport model
+
+pyserial 3.5 is installed at:
+
+```text
+/usr/lib/python3/dist-packages/serial
+```
+
+Its architecture:
+
+- common `SerialBase` plus platform backends;
+- POSIX backend uses `termios`, `fcntl` and `select`/`poll`;
+- Windows backend uses the Win32 serial API;
+- URL handlers for `socket://`, `rfc2217://`, `loop://`, `spy://`;
+- port enumeration via `list_ports` and sysfs metadata;
+- threaded reader/protocol adapters and separate asyncio integration.
+
+Useful API semantics:
+
+- separate read, write and inter-byte timeouts;
+- blocking, non-blocking and infinite timeout modes;
+- short reads on timeout and partial-write handling;
+- `in_waiting` / `out_waiting`;
+- `cancel_read` / `cancel_write`;
+- input/output buffer reset;
+- `flush` waits for transmission to drain;
+- DTR/RTS/CTS/DSR/RI/CD;
+- XON/XOFF and RTS/CTS flow control;
+- `send_break`;
+- exclusive access via `flock`;
+- custom baud via `BOTHER`, low-latency mode, RS485 on Linux;
+- loopback/spy transports for tests.
+
+Comparison:
+
+| Area | TopSpeed | pyserial |
+| --- | --- | --- |
+| Target | DOS/OS-2 PC UART | POSIX/Windows OS tty |
+| Serial layer | direct UART + IRQ | termios + OS driver + poll |
+| Buffering | application ring buffers | kernel buffers + waiting counts |
+| Timeouts | tenths, busy wait | read/write/inter-byte, monotonic |
+| Cancellation | none | read/write cancellation |
+| Multiple ports | one active singleton | multiple instances |
+| Discovery | COM1–COM4 | device paths, URLs, sysfs |
+| Advanced | hooks | RS485, custom baud, low latency, exclusive |
+| Protocols | X/Y/WX/Kermit | none in core |
+
+Recommendation: model V3’s serial layer on pyserial’s feature set, not
+on TopSpeed’s hardware driver.  Keep TopSpeed’s protocol descriptions
+and state machines as references for optional file-transfer modules.
+
+### 11.4 Recommended communications architecture
+
+Core:
+
+```text
+Serial       POSIX tty transport
+Transport    read/write/flush/poll/cancel/close interface
+Socket       optional TCP transport
+Loopback     deterministic test transport
+```
+
+Protocols and application layers:
+
+```text
+XModem / YModem / WXModem   optional
+Kermit                     only if required
+Terminal                   optional VT100/ANSI emulation
+Script                     optional application-level VM
+```
+
+The protocol modules must depend only on `Transport`, never directly on
+`Serial`.
+
+A C shim should implement the POSIX details:
+
+```text
+m2serial_open
+m2serial_configure
+m2serial_read
+m2serial_write
+m2serial_poll
+m2serial_flush
+m2serial_set_modem
+m2serial_send_break
+m2serial_close
+```
+
+Use status codes, not fatal errors.  Support:
+
+- blocking and non-blocking modes;
+- separate read/write/inter-byte timeouts;
+- cancellation;
+- `in_waiting`/`out_waiting`;
+- flow control;
+- modem lines;
+- exclusive open;
+- optional custom baud/RS485/low-latency;
+- loopback tests.
+
+A blocking read with timeout is sufficient initially.  A reader
+coroutine or event loop can be added later; avoid binding the API to
+threads.
+
+### 11.5 ModBase
+
+References:
+
+```text
+$HOME/bin/Dos/M2/TS-V3/MODBASE/MODBASE3.DEF
+$HOME/bin/Dos/M2/TS-V3/MODBASE/DBINDXES.DEF
+$HOME/bin/Dos/M2/TS-V3/MODBASE/DBFIELDS.DEF
+$HOME/bin/Dos/M2/TS-V3/MODBASE/LOCKS.DEF
+$HOME/bin/Dos/M2/TS-V3/MODBASE/HASH.DEF
+$HOME/bin/Dos/M2/TS-V3/Manuals/Modbase 3.odt
+```
+
+ModBase is a dBase III-compatible database engine:
+
+- `.DBF` data files, `.NDX` indexes, `.DBT` memo files;
+- fixed/variable fields;
+- multiple indexes per data file;
+- automatic index updating via an update list;
+- buffered record and index I/O;
+- safety/flush modes;
+- record/range locking with retries;
+- delete/undelete flags;
+- index rebuild/check tools;
+- full-screen field editing via Repertoire.
+
+Core modules:
+
+```text
+ModBase3   data file, record buffer, fields, memo, locks
+DBIndxes   B-tree index, cursors, build/rebuild, auto update
+DBFields   typed field access (date, logical, numeric, string)
+DBRelate   relational lookup through an index
+Locks      file/range locking and retry
+```
+
+`DBIndxes` uses 512-byte B-tree nodes, a configurable in-memory node
+buffer, a 128-byte key limit, and a maximum depth of 24.  `Hash` is an
+opaque byte-key table from Repertoire, not part of the core dBase
+engine.
+
+Good ideas to borrow:
+
+- opaque `DBFile`/`DBIndex` handles;
+- explicit schema/field descriptors;
+- typed accessors over a raw record buffer;
+- multiple secondary indexes and an update list;
+- key-derivation callbacks for composite indexes;
+- explicit safety/flush policy;
+- range/record locking with retry policy;
+- rebuild/check utilities;
+- cursor operations (`GoTop`, `GoBottom`, `Next`, `Prev`);
+- separate memo storage;
+- `Hash` API shape for a generic hash table.
+
+Why not port it:
+
+- it is a dBase III compatibility project, not a general database layer;
+- it depends on Repertoire modules (`HandleIO`, `StringIO`, `LowLevel`,
+  `VStorage`, `ErrorManager`, `SmartScreen`, `FAPI`) and DOS/OS-2
+  services;
+- compiler-independent initialization and per-compiler number handling
+  add indirection V3 does not need;
+- error handling is mostly integer OS codes plus interactive
+  confirmation;
+- no transaction/WAL story;
+- dBase III storage/index formats are legacy;
+- the license restricts source/unlinked-object redistribution.
+
+Recommendation:
+
+- use SQLite through a C binding for general database needs;
+- or build a small typed table/index layer on `BTreeFile`;
+- implement dBase III compatibility only if it is an explicit
+  requirement, as a separate project with compatibility tests;
+- use ModBase’s schema/index/locking ideas as design input only.
+
+### 11.6 Suggested order within communications/database work
+
+1. `Serial` POSIX transport with pyserial-inspired semantics.
+2. `Transport` abstraction plus `Loopback` tests.
+3. Optional `Socket` transport.
+4. Optional XMODEM/YMODEM.
+5. Optional terminal emulation.
+6. Optional script VM.
+7. Database: SQLite binding or typed B-tree table layer.
+8. dBase III/ModBase compatibility only if required.
+
+---
+
+## 12. Suggested order of future work
 
 1. `TBITSIZE` (small, standard-facing).
 2. `ADDADR`/`SUBADR`/`DIFADR` (small, ISO SYSTEM).
@@ -1059,9 +1347,13 @@ The TopSpeed B-tree Toolkit should not delay steps 1–3.
    and UString modules, then UTF-8 input.
 7. Collections: `Vector`, `Deque`/`Stack`, `HashMap`, then optional
    `BTreeMap`; persistent `BTreeFile` last.
-8. ISO `COROUTINES` subset, then interrupt/protection layer.
-9. Dynamic `SET CONST` and deep composite copies.
-10. Parser error recovery and diagnostics.
+8. Communications: `Serial`/`Transport` first; protocols and terminal
+   emulation optional.
+9. Database: SQLite binding or typed B-tree table layer; dBase III only
+   if required.
+10. ISO `COROUTINES` subset, then interrupt/protection layer.
+11. Dynamic `SET CONST` and deep composite copies.
+12. Parser error recovery and diagnostics.
 
 External assembly needs no compiler work; document it as the supported
 low-level escape hatch.