docs: analyse collections and TopSpeed B-tree toolkit

docs/INDEX.md

@@ -36,8 +36,8 @@ bisecting *why* a decision was made, not for reading end-to-end).
 
 - **`future-work-2026-10-25.md`** — TBITSIZE, ADDRESS arithmetic, set
   ordering, complex types, pragmas (TopSpeed vs portable), UTF-8 string
-  libraries, PIM vs ISO coroutines, external assembly, dynamic-set
-  design and suggested order.
+  libraries, collections and the TopSpeed B-tree Toolkit, PIM vs ISO
+  coroutines, external assembly, dynamic-set design and suggested order.
 
 ## Designs / running records
 

docs/future-work-2026-10-25.md

@@ -3,8 +3,9 @@
 This note records the design discussions from 2026-10-25 so the
 questions do not have to be re-derived later.  It covers `TBITSIZE`,
 address arithmetic, set ordering, complex types, pragmas, UTF-8 string
-libraries, PIM versus ISO coroutines, external assembly, and the
-dynamic-set design.  It is not a commitment to implement everything.
+libraries, collections, the TopSpeed B-tree Toolkit, PIM versus ISO
+coroutines, external assembly, and the dynamic-set design.  It is not a
+commitment to implement everything.
 
 **State at the time of writing:** suite 310/310, one-shot fixpoint OK
 (3,845,439 bytes), per-object fixpoint OK (11 objects), corpus
@@ -862,7 +863,190 @@ These need generated Unicode tables, not hand-written ranges.
 
 ---
 
-## 10. Suggested order of future work
+## 10. Collections library and the TopSpeed B-tree Toolkit
+
+### 10.1 Current state
+
+V3 has no collections library.  What exists is only infrastructure:
+
+- `lib/common/Storage` — `ALLOCATE`, `DEALLOCATE`, `REALLOCATE`
+- `lib/gm2/SysStorage` — raw allocation
+- `lib/gm2/DynamicStrings` — heap strings
+- ISO generics: type/value/procedure parameters and kind constraints
+- generic container demonstrations in tests:
+  - `compiler/tests/showcase23lib.def` — `DList(T)`
+  - `compiler/tests/showcase24stack.def` — `GStack(T)`
+
+Those test modules are proof of concept, not production libraries: they
+have no `Done`/`Clear`, no ownership policy, and use one `NEW` per node.
+The `MyWork/Strings/DataStructures` FIFO/LIFO/dynamic-array files are
+incomplete and are not part of V3.
+
+### 10.2 What the TopSpeed B-tree Toolkit is
+
+The real toolkit is installed with the TS-V3 tree.  Useful references:
+
+```text
+$HOME/bin/Dos/M2/TS-V3/INCLUDE/BTREE.DEF
+$HOME/bin/Dos/M2/TS-V3/SRC/BTREE.MOD
+$HOME/bin/Dos/M2/TS-V3/DOC/BTREE.DOC
+$HOME/bin/Dos/M2/TS-V3/Manuals/B-Tree toolkit.odt
+```
+
+It is a **disk-based B-tree/ISAM toolkit**, not an in-memory collections
+library.  Its architecture is:
+
+- one physical file (`FHandle`)
+- multiple logical data and index files (`IHandle`) in that physical file
+- fixed or variable-length data records, up to 65,535 bytes
+- index keys up to 128 bytes
+- multiple indexes per data file, updated automatically by `Add`/`Delete`
+- application-supplied `CompareFunction` and `KeyFunction`
+- duplicate keys configurable per index
+- `Find`/`Search` (exact and greater-or-equal), `Next`/`Prev`/`Reset`
+  cursors
+- `Add`, `Delete`, `Change`, `Open`, `OpenData`, `OpenIndex`, `Close`,
+  `Flush`, `LastError`, `LastFError`
+- 1024-byte pages with a 32-page LRU cache
+- optional data compression through `Pack`/`APACK2`
+- shared-file locking and multi-tasking support
+- 4 GiB physical-file limit
+- free-list allocation; the file never shrinks automatically
+- no automatic compaction
+- default error handler prints and terminates the program
+
+The current definition also includes `AccessMode` values such as
+`FixSize`, `Size16`, `Size32`, `NoDealoc` and `Compress`.
+
+The source is 16-bit DOS/OS/2 code: inline x86 assembly, segment
+arithmetic, TopSpeed pragmas, `FIOx`, `CoreMem`, and raw
+`ADDRESS`/`ARRAY OF BYTE` interfaces.  It is a design reference, not a
+portable V3 implementation.
+
+### 10.3 Good ideas to borrow
+
+- Separate logical data and index files, with multiple indexes per data
+  file.
+- Application-supplied key derivation and comparison.
+- Cursor-oriented API (`Find`, `Search`, `Next`, `Prev`, `Reset`) rather
+  than exposing pages.
+- Per-handle error state, with an optional handler.
+- Bounded page cache with LRU replacement.
+- Configurable duplicate-key handling.
+- Explicit open/close/flush lifecycle.
+- A versioned file header and free-space management.
+- Optional compression as a separate layer.
+
+These map naturally to V3 generics:
+
+```modula2
+Compare : PROCEDURE(a, b : K) : INTEGER;
+Hash    : PROCEDURE(k : K) : CARDINAL;
+Equal   : PROCEDURE(a, b : K) : BOOLEAN;
+```
+
+### 10.4 What not to copy
+
+- Fatal-error-by-default: a library must return status.
+- Raw `ADDRESS`/`ARRAY OF BYTE` public interfaces: provide typed
+  wrappers.
+- Fixed 128-byte keys, 1 KiB pages and 4 GiB files: make limits
+  explicit/configurable where practical.
+- 16-bit segment/assembly code.
+- DOS `SHARE`/`FIOx` locking: use V3 file APIs and POSIX locking later.
+- No compaction: provide a copy/rebuild tool or document the limitation.
+- Raw binary records without schema/version metadata.
+- Automatic indexing only for future additions; rebuilding an index
+  must be an explicit operation.
+- Copying the source code: it is old, platform-specific and carries its
+  own licensing terms.
+
+### 10.5 Recommended library layout
+
+Keep in-memory collections and the persistent B-tree separate.
+
+**In-memory containers**
+
+```text
+Vector(T)       dynamic array
+Deque(T)        FIFO / double-ended queue
+Stack(T)        LIFO
+HashMap(K,V)    hash table with Hash/Equal callbacks
+BTreeMap(K,V)   optional ordered map/set with Compare
+```
+
+`Vector` is the foundation.  `Stack` and `Deque` should reuse it or a
+ring-buffer implementation.  Avoid one allocation per element unless
+stable references are required.
+
+`HashMap` should use open addressing with tombstones or separate
+chaining, a load factor around 0.7, and explicit resize/rehash.  Add a
+`StringMap(V)` convenience wrapper with owned string keys.
+
+`BTreeMap` is the in-memory ordered structure: comparator callback,
+first/last/next/prev, range scan, optional duplicate keys, no pager.
+
+**Persistent B-tree**
+
+```text
+BTreeFile  — disk-based index, toolkit-inspired
+```
+
+This is a separate library, not part of `Vector`/`HashMap`.  It needs:
+
+- file format and versioning
+- page size and cache policy
+- split/merge and free-space management
+- cursor state
+- locking policy
+- crash/durability policy
+- compaction/rebuild
+- optional compression
+
+The TopSpeed toolkit is the checklist for this stage.
+
+### 10.6 V3 constraints
+
+- No GC: containers need explicit `Init`/`Done`/`Clear`.
+- No automatic deep copy: pointer-element ownership must be documented.
+- `Storage.REALLOCATE` exists, but arbitrary pointer arithmetic does
+  not.  TopSpeed uses `AddAddr`/`IncAddr`/`DecAddr` assembly helpers;
+  V3 cannot rely on that.
+- A truly growable `Vector(T)` needs one of:
+  - a fixed maximum capacity parameter,
+  - chunked storage,
+  - or a small compiler/runtime extension for variable-sized typed
+    arrays.
+- Generic instances are supported, but multiple actual-type instances
+  should be regression-tested.
+- Keep array length and capacity separate.
+- Use records and modules, not classes, for containers.
+- Do not use a fatal error handler as the default.
+
+Suggested error model:
+
+```modula2
+TYPE Status = (Ok, Empty, NotFound, InvalidIndex,
+               OutOfMemory, DuplicateKey);
+```
+
+If enum handling is awkward in a given context, use integer status codes
+or `BOOLEAN` plus `LastError`.
+
+### 10.7 Suggested order within the collections work
+
+1. `Vector(T)`
+2. `Deque(T)` / `Stack(T)`
+3. `HashMap(K,V)`
+4. `StringMap(V)`
+5. In-memory `BTreeMap(K,V)`
+6. Persistent `BTreeFile` based on the TopSpeed architecture
+
+The TopSpeed B-tree Toolkit should not delay steps 1–3.
+
+---
+
+## 11. Suggested order of future work
 
 1. `TBITSIZE` (small, standard-facing).
 2. `ADDADR`/`SUBADR`/`DIFADR` (small, ISO SYSTEM).
@@ -873,9 +1057,11 @@ These need generated Unicode tables, not hand-written ranges.
    subset; later add the TopSpeed adapter and ABI/layout pragmas.
 6. UTF-8 string/character libraries: fix `Utf8.Encode`, add byte-string
    and UString modules, then UTF-8 input.
-7. ISO `COROUTINES` subset, then interrupt/protection layer.
-8. Dynamic `SET CONST` and deep composite copies.
-9. Parser error recovery and diagnostics.
+7. Collections: `Vector`, `Deque`/`Stack`, `HashMap`, then optional
+   `BTreeMap`; persistent `BTreeFile` last.
+8. ISO `COROUTINES` subset, then interrupt/protection layer.
+9. Dynamic `SET CONST` and deep composite copies.
+10. Parser error recovery and diagnostics.
 
 External assembly needs no compiler work; document it as the supported
 low-level escape hatch.