Future-work analysis — 2026-10-25

This note records the design discussions from 2026-10-25 so the questions do not have to be re-derived later. It covers TBITSIZE, address arithmetic, set ordering, complex types, pragmas, UTF-8 string libraries, collections, the TopSpeed B-tree Toolkit, PIM versus ISO coroutines, external assembly, and the dynamic-set design. It is not a commitment to implement everything.

State at the time of writing: suite 310/310, one-shot fixpoint OK (3,845,439 bytes), per-object fixpoint OK (11 objects), corpus 521/612 compile-OK, 0 crashes, 64 LANG and 27 LIB.

1. `TBITSIZE`

TBITSIZE is a GNU Modula-2 SYSTEM procedure function:

PROCEDURE TBITSIZE (<type>) : CARDINAL;

It returns the minimum number of bits needed to represent a type, not its normal storage size in bytes. It is mainly useful for packed records/bitfields. A BOOLEAN may be one bit logically but occupy a word in an unpacked record.

GNU Modula-2 supports it (and uses it in M2WIDESET).
V3 does not support it.
V3's existing SIZE/TSIZE report storage size in bytes.
Adding it would need:
- a type/variable argument form in the grammar;
- a descriptor query for minimum bit width (integer/real/char/enum/ subrange/record/packed-record cases);
- constant folding so TBITSIZE(BYTE) = 8 works in CONST;
- interaction with packed record layout.

It is a small-to-medium language gap, not a runtime feature.

2. `ADDRESS` arithmetic

ISO SYSTEM defines:

PROCEDURE ADDADR (addr: ADDRESS; offset: CARDINAL): ADDRESS;
PROCEDURE SUBADR (addr: ADDRESS; offset: CARDINAL): ADDRESS;
PROCEDURE DIFADR (addr1, addr2: ADDRESS): INTEGER;
PROCEDURE MAKEADR (high: <type>; ...): ADDRESS;

V3 currently supports:

ADR
ADDRESS/pointer assignment
ADDRESS/pointer equality
VAL/CAST for compatible scalar representations

V3 does not support:

ADDADR/SUBADR/DIFADR/MAKEADR
ADDRESS + offset, ADDRESS - offset, ADDRESS - ADDRESS
C-style arithmetic on typed POINTER TO values

Why it is missing

ADDRESS is an opaque pointer-like value in V3's type system. Raw pointer arithmetic is not part of the PIM/ISO base language and would weaken V3's separation between typed pointers and addresses.

Recommended implementation

Add the ISO SYSTEM functions first, not general pointer arithmetic:

byte-addressed ADDADR/SUBADR;
DIFADR returning the byte difference (range-checked for INTEGER);
optional MAKEADR;
tests for offsets, null addresses, overflow and pointer/address equality.

For external C/assembly interaction, a C helper remains the pragmatic escape hatch.

3. Set ordering

PIM/ISO set operations are:

equality =, inequality #, membership IN;
union +, difference -, intersection *, symmetric difference /.

There is no standard <, <=, >, >= ordering for sets. GNU Modula-2 adds inclusion ordering:

a <= b   (* a is a subset of b *)
a <  b   (* a is a proper subset of b *)
a >= b   (* a is a superset of b *)
a >  b   (* a is a proper superset of b *)

V3 currently rejects these; corpus case largeset5 remains a LANG failure.

If implemented as a gm2-compatibility extension

Fixed/inline sets:

a <= b   iff   (a AND NOT b) == 0 for every word
a <  b   iff   a <= b AND a # b

Dynamic sorted-range sets:

a <= b   iff every range of a is covered by a range of b
a <  b   iff a <= b AND a # b

Compiler work:

SymTab.RelCheck/OrdCheck: accept set ordering;
Lower: branch fixed/dynamic set ordering;
QbeGen: subset/superset comparisons;
tests: empty sets, equal sets, subset, proper subset, superset, disjoint sets, mixed widths;
largeset5 would likely move from LANG to compile-OK.

This is a safe gm2 compatibility extension, not standard PIM/ISO.

4. PIM versus ISO coroutines

PIM

Low-level SYSTEM primitives:

TYPE PROCESS;
PROCEDURE NEWPROCESS (body: PROC; workspace: ADDRESS;
                      size: CARDINAL; VAR p: PROCESS);
PROCEDURE TRANSFER (VAR from: PROCESS; to: PROCESS);
PROCEDURE IOTRANSFER (VAR from, to: PROCESS;
                      interruptNo: CARDINAL);

caller supplies workspace;
caller explicitly transfers;
no scheduler;
no coroutine identity query;
no attach/detach/protection model;
IOTRANSFER uses an explicit interrupt number.

V3 implements this with ucontext; IOTRANSFER currently degrades to a normal transfer because there is no interrupt subsystem.

ISO 10514-1 `COROUTINES`

DEFINITION MODULE COROUTINES;

TYPE
  COROUTINE;
  INTERRUPTSOURCE = CARDINAL;
  PROTECTION = [UnassignedPriority .. 7];

PROCEDURE NEWCOROUTINE (procBody: PROC; workspace: ADDRESS;
                        size: CARDINAL; VAR cr: COROUTINE;
                        [initProtection: PROTECTION]);
PROCEDURE TRANSFER (VAR from: COROUTINE; to: COROUTINE);
PROCEDURE IOTRANSFER (VAR from: COROUTINE; to: COROUTINE);
PROCEDURE ATTACH (source: INTERRUPTSOURCE);
PROCEDURE DETACH (source: INTERRUPTSOURCE);
PROCEDURE IsATTACHED (source: INTERRUPTSOURCE): BOOLEAN;
PROCEDURE HANDLER (source: INTERRUPTSOURCE): COROUTINE;
PROCEDURE CURRENT (): COROUTINE;
PROCEDURE LISTEN (p: PROTECTION);
PROCEDURE PROT (): PROTECTION;
PROCEDURE TurnInterrupts (to: PROTECTION): PROTECTION;
PROCEDURE ListenLoop;
END COROUTINES.

Main differences:

Area	PIM	ISO
Context type	`PROCESS`	`COROUTINE`
Creation	`NEWPROCESS`	`NEWCOROUTINE` + optional protection
Switch	`TRANSFER`	`TRANSFER`
Interrupt transfer	`IOTRANSFER(..., interruptNo)`	`IOTRANSFER(from,to)` with attached source
Current context	none	`CURRENT`
Interrupt registration	none	`ATTACH`/`DETACH`/`IsATTACHED`/`HANDLER`
Protection/priority	none	`PROTECTION`, `LISTEN`, `PROT`, `TurnInterrupts`
Event loop	user-written	`ListenLoop`

V3 has PIM coroutines but no ISO COROUTINES module. A pure coroutine subset (NEWCOROUTINE, TRANSFER, CURRENT, workspace management) could map onto the existing ucontext layer. The real interrupt/protection parts need an OS signal/event layer.

5. External assembly instead of inline `ASM`

External assembly works today through EXTERNAL and the normal linker; no compiler support for inline ASM is required.

DEFINITION MODULE AsmOps;
PROCEDURE Add (a, b : CARDINAL) : CARDINAL;
END AsmOps.

IMPLEMENTATION MODULE AsmOps;
PROCEDURE Add (a, b : CARDINAL) : CARDINAL;
  EXTERNAL;
END AsmOps.

    .text
    .globl Add
    .type Add,@function
Add:
    movl %edi, %eax
    addl %esi, %eax
    ret

cc -c asm_add.s -o asm_add.o
M2 -c AsmOps.def AsmOps.mod
qbe -o AsmOps.s AsmOps.ssa
cc -c AsmOps.s -o AsmOps.o
cc Prog.o AsmOps.o asm_add.o runtime/syslib/shim.c -o Prog -lm

For M2make --separate, add the assembly object with --extra asm_add.o or link manually.

ABI notes

QBE target is amd64_sysv.
INTEGER/CARDINAL are 32-bit; LONGINT is 64-bit.
pointers are 64-bit l values.
VAR parameters receive addresses.
external procedures get no V3 static-link argument.
open arrays, especially ARRAY OF CHAR, are passed as descriptor addresses, not raw C arrays.
record/array value parameters are passed by address.
composite results use a hidden result pointer.

For anything beyond scalar arguments, use a small C wrapper around the assembly routine so the V3-facing signature is C-compatible.

This is the recommended route for low-level code; inline ASM is unnecessary.

6. Dynamic `SET OF CARDINAL` / `INTEGER` — chosen design

V3 now uses a sorted linked list of disjoint inclusive ranges:

set = pointer to
  { lo : CARDINAL; hi : CARDINAL; next : pointer }

This was chosen over a full 2^32-bit bit-vector (512 MiB per set value) and over a linked list of individual bits (too much per-value overhead). The range-list representation is:

O(number of contiguous runs), not O(cardinality);
compact for dense ranges;
supports the full 32-bit base range;
INTEGER values map with VAL(CARDINAL, x) XOR 0x80000000.

Implemented operations include empty/range literals, INCL/EXCL/IN, assignment, equality, union/intersection/difference and symmetric difference. Bounded sets such as SET OF [0..100000] keep the inline bit-vector form.

Current limitations

dynamic SET CONST is rejected (loud 230), because no static range-list layout exists yet;
record/array assignment containing dynamic sets currently copies the pointer rather than deeply copying the range list;
nodes use malloc; assignment frees old lists but there is no GC or program-exit cleanup;
PACKEDSET remains capped at 32 bits.

Future work: runtime-initialised dynamic constants, deep composite copy, ownership/lifetime rules, and optionally a hash/paged bitset if range lists become too fragmented.

7. Complex types: built-in types versus a library

V3 currently has no complex arithmetic. COMPLEX32, COMPLEX64 and COMPLEX128 are declaration-compatible aliases of REAL (currently 8 bytes each), so they are placeholders, not real complex types. Their names imply two 4-, 8- and 16-byte components (8, 16 and 32 bytes respectively), so the current aliases are also the wrong size. The comparison table correctly marks COMPLEX/LONGCOMPLEX as unsupported.

What ISO expects

ISO Modula-2 treats COMPLEX/LONGCOMPLEX as predefined types. The language is expected to provide:

complex arithmetic operators (+, -, *, /);
the CMPLX constructor;
RE/IM accessors;
complex-aware assignment, parameters and function results.

Transcendental functions live in the separate ComplexMath and LongComplexMath library modules. The ISO architecture is therefore:

compiler-known COMPLEX/LONGCOMPLEX types
        + operator lowering / builtins
        + ComplexMath / LongComplexMath libraries

A separate library can provide complex-number functions, but it cannot by itself provide natural Modula-2 syntax such as a + b or CMPLX. Operator overloading does not exist in Modula-2, so a library-only design is necessarily a nonstandard utility API.

Why not classes

Classes are a poor fit for complex numbers:

complex numbers are small value types, not objects with identity;
class methods give a.Add(b) instead of a + b;
class allocation/dispatch is unnecessary overhead for a pair of reals;
classes are not ISO Modula-2 OO (V3's CLASS is Clarion-form);
a class still cannot add operators, literals or RE/IM syntax.

Use a value record or a compiler-known type, not a class.

What generics can do

V3 has ISO 10514-2 generics, so a generic implementation could share code between real and long-real variants:

GENERIC MODULE Complex(T);
TYPE Value = RECORD re, im : T END;
...
END Complex.

That is useful for implementation sharing, but generics still do not provide operators, literals or ISO source compatibility. Since V3's REAL and LONGREAL are both 8 bytes today, two small concrete modules are also acceptable and simpler for separate compilation.

Option A: nonstandard value-record library

For practical complex math without an ISO-conformance claim:

DEFINITION MODULE Complex;

TYPE
  T = RECORD
        re, im : REAL
      END;

PROCEDURE Add (a, b : T) : T;
PROCEDURE Sub (a, b : T) : T;
PROCEDURE Mul (a, b : T) : T;
PROCEDURE Div (a, b : T) : T;
PROCEDURE Conj (z : T) : T;
PROCEDURE Abs (z : T) : REAL;
PROCEDURE Arg (z : T) : REAL;

END Complex.

Advantages: value semantics, no allocation, simple ABI, easy to test. Limitations: call syntax only; no operator or literal support; does not satisfy ISO COMPLEX.

Do not call this type COMPLEX or reuse the existing COMPLEX* aliases, because that would falsely suggest ISO support.

Option B: ISO-compatible built-in types

Implement COMPLEX/LONGCOMPLEX as compiler-known value types, with arithmetic lowered to runtime helpers:

COMPLEX = two REAL values
a + b   -> __m2_complex_add(a, b)
a * b   -> __m2_complex_mul(a, b)
CMPLX   -> compiler builtin
RE/IM   -> compiler builtins

Then implement ComplexMath and LongComplexMath as libraries on top. This avoids requiring QBE to understand a native complex type while still providing ISO syntax and semantics.

Work items:

representation/layout (COMPLEX is 16 bytes with the current 8-byte REAL; COMPLEX32 cannot be 8 bytes until a real 4-byte SHORTREAL exists);
type checking for mixed real/complex expressions and conversions;
lowering of arithmetic to helper calls;
CMPLX, RE, IM and constant expressions;
ABI for complex parameters/results, including separate compilation;
library implementations of abs, arg, conj, sqrt, exp, ln, trigonometric functions and polarToComplex;
tests for arithmetic, constants, mixed precision, copying, parameters, function results and separate objects.

For C interop, do not assume the C99 _Complex ABI without checking; prefer small C wrappers where necessary.

Recommendation

If ISO conformance is a project goal, complex types must eventually be built-in; the library is the math layer, not a replacement.
If only practical complex math is wanted, add the value-record Complex library first, with no classes and no fake COMPLEX names.
Generics are optional implementation sharing, not a substitute for language support.
Regardless of option, remove or clearly mark the current COMPLEX32/64/128 REAL aliases so programs cannot silently get wrong-size, non-complex values.

8. Pragmas: TopSpeed compatibility versus portable scoped pragmas

V3 has no general pragma facility. It currently parses and ignores two gm2 declaration pragmas (__BUILTIN__ and __ATTRIBUTE__ __BUILTIN__ (( name ))) after PROCEDURE, and its TopSpeed sidecar lexes (*# ... *) as an ordinary comment. The local EXCEPTIONS.def therefore omits <* noreturn *>.

There are two separate design questions:

give V3 a native pragma facility;
consume legacy TopSpeed pragmas.

The recommendation is to separate them: use the portable, scoped model as the native design, and add a narrow TopSpeed compatibility mode.

8.1 TopSpeed V3 pragmas

TopSpeed Modula-2 pragmas are a special form of comment:

(*# save *)
(*# check(range => on, index => off, overflow => off) *)
PROCEDURE P (...);
(*# restore *)

They are stateful compiler directives:

a setting applies from that point forward;
save / restore maintain a stack of compiler settings;
a procedure's calling convention is the setting in force at its definition;
a variable's layout is the setting in force at its declaration.

They are grouped into pragma classes:

check(...) — range, index, overflow, NIL, stack checks;
call(...) — calling convention, register parameters, interrupt handlers, open-array passing;
data(...) — segments, packing, volatile, enum size, stack/heap size;
optimize(...) — CSE, register allocation, peephole, CPU choice;
module(...) — initialization code, implementation-object rules;
name(...) — external symbol naming;
debug(...) — debug information and procedure tracing;
option(...) — language extensions such as bitwise operations on cardinals;
save / restore — compiler-option stack;
link, project, define — project-system integration.

Older $ compiler directives are also accepted by TopSpeed for compatibility.

Strengths

Pragmatic and expressive.
Local check overrides without recompiling the build system.
Historical TopSpeed code already uses the syntax.
Easy to lex as a comment; V3's corpus already parses (*# ... *).

Problems for V3

Much of it is tied to 16-bit x86: segments, near/far calls, interrupts, reg_param, iopl, memory models, overlays, CPU and coprocessor selection. These do not map to V3's amd64/QBE target.
Semantics are position/state-dependent rather than attached to declarations.
Source annotations are mixed with global build settings such as optimisation and debug options.
It is a frozen, vendor-specific dialect, not a portable standard.
Because (*# ... *) is a comment to other compilers, ignoring it silently changes checks, layout or ABI without any diagnostic.

TopSpeed pragmas are therefore useful as a compatibility adapter, not as V3's native pragma model.

8.2 Benjamin / R10 portable pragmas

The portable proposal (used by Modula-2 R10/BSK) uses <* ... *>:

PROCEDURE Reboot <*NORETURN*>;

TYPE Header = RECORD <*PACKED*>
  ...
END;

VAR Port <*VOLATILE*> : BYTE;

It defines:

standard pragmas in uppercase;
implementation-defined pragmas with a lowercase implementation prefix, e.g. <*v3.SomeOption=...*>;
explicit scopes for module, type, record, field, procedure and parameter placement, plus point-forward and no-scope placement;
combinable clauses in one block;
PRESETS / UNSET for external pragma settings instead of a TopSpeed-style global save / restore;
a mandatory BSK subset: DEPRECATED, FFI, FFIDENT, INLINE and PRIVATETO, plus ENCODING where Unicode is supported.

The full proposal is much larger: ABI, ADDR, ALIGN, PACKED, PADBITS, LEADBITS, VOLATILE, NORETURN, PURE, WEAK, IN, OUT, SINGLEASSIGN, ARC, GC, OWNER, BORROWER, RETAIN, RELEASE, MSG, TELL, TICKET and others. R10 itself omits some of these (RANGE, RELEASE, RETAIN).

Strengths

Attached to syntax, not a global mutable state.
Deterministic and easier to validate.
Fits definition/implementation consistency and separate compilation.
<* ... *> aligns with ISO-style/gm2 pragma syntax.
Implementation-defined pragmas have a collision-free namespace.
Provides a path to standard annotations for ABI, layout, volatility and diagnostics.
Much better fit for a strict, self-hosting compiler.

Problems

The proposal is large; implementing all of it is a project of its own.
Many pragmas are promises that require real analysis: PURE/WEAK/DETM need effect checking, IN/OUT need parameter-write analysis, NORETURN needs control-flow checks and SINGLEASSIGN needs data-flow checks.
Some are irrelevant to V3: ARC, GC, OWNER, BORROWER, RETAIN, RELEASE; RESIDENT, CONTIGUOUS and LOWLATENCY are mostly OS/runtime hints; ENCODING needs scanner/Unicode work.
ADDR, ALIGN, PACKED, PADBITS and LEADBITS require real layout support.
The proposal is still evolving; V3 should implement a versioned subset, not chase every edit.

8.3 Comparison

Aspect	TopSpeed V3	Benjamin / portable
Status	Vendor dialect	Proposed portable specification, used by R10/BSK
Delimiters	`(# ... )`	`<* ... *>`
Form	`class(name => value, ...)`, `save`, `restore`	`PRAGMA`, `PRAGMA=expr`, clause sequences
Scope	Stateful from point forward	Attached to explicit syntactic scopes
ABI/layout	Many 16-bit-specific directives	Portable ABI/layout annotations
Unknown pragmas	Often ignored or comment-like	Standard versus implementation-defined, diagnosable
Separate compilation	Needs external project-state discipline	Naturally attachable to symbols and object stamps
V3 fit	Compatibility adapter	Native pragma design

TopSpeed is not a standard; the <* ... *> syntax is the better long-term direction because it is already used by ISO-style/gm2 compilers and by the portable proposal.

8.4 Recommended V3 design

Use <* ... *> with explicit scope attachment as the native pragma syntax. Do not make (*# ... *) the default; handle it only in the TopSpeed sidecar or behind an explicit compatibility flag.

Phase 1 — infrastructure and low-risk semantics

scanner/parser support for <* ... *> and adjacent blocks;
AST/symbol-table pragma nodes with source locations;
scope validation;
unknown-pragma policy;
DEPRECATED;
NORETURN;
NOINLINE;
INLINE as advisory, with a warning if not honoured;
MSG / TICKET diagnostics if desired;
PRIVATETO.

NORETURN is especially valuable immediately: it lets V3 consume standard ISO/gm2 definition files without stripping pragmas.

Phase 2 — layout and ABI

PACKED;
ALIGN / MEMALIGN;
FFI / FFIDENT;
ADDR only once absolute variables and address arithmetic are designed;
PADBITS / LEADBITS only after packed-record layout exists.

Every ABI/layout pragma must participate in:

definition/implementation consistency checks;
the separate-compilation object ABI stamp;
the content hash used by M2make --separate.

Otherwise two objects can disagree about record layout or symbol names and still link.

Phase 3 — contracts

PURE, WEAK, DETM;
IN / OUT;
SINGLEASSIGN.

These should not be accepted as no-ops. If V3 cannot verify them, it should reject them or emit a clear "accepted but not enforced" warning. Silently pretending to enforce a promise is worse than not supporting it.

Do not implement initially

ARC, GC, OWNER, BORROWER, RETAIN, RELEASE;
RESIDENT, CONTIGUOUS, LOWLATENCY;
full PRESETS / pragma-file machinery;
Unicode ENCODING.

These need runtime support V3 does not have, or are platform hints with little value on the current target.

8.5 TopSpeed compatibility

Keep TopSpeed pragmas out of the default V3 grammar. Handle them in the existing TopSpeed-V3-M2.atg sidecar, or add a --topspeed-pragmas mode to the main compiler.

Map only the meaningful subset:

check(range/index/overflow/nil/stack);
data(packed/volatile);
module(init_code);
call(c_conv);
save / restore.

Reject or warn about:

near_call, far, segments, seg_name, ds_*;
reg_param, reg_saved, reg_return;
interrupt, iopl;
overlays, memory models, cpu, copro;
Windows/OS2/DOS-specific options.

TopSpeed stateful pragmas can be translated by keeping an option stack during parsing and snapshotting the current state onto each declaration. That is an adapter, not the native model.

8.6 Never silently ignore

A TopSpeed pragma that V3 treats as a comment must produce at least a warning in TopSpeed-compatibility mode.
An unknown standard pragma should be an error.
An unknown v3.* pragma should be a warning or informational message.

Otherwise V3 will compile code with different checks or ABI and call it success.

8.7 Sources

TopSpeed Developer's Guide, chapter on pragmas.
trijezdci/M2-Pragmas wiki, especially Pragma Specification, Pragma Placement and Pragma Scopes.
Modula-2 R10/BSK pragma facilities chapter.

9. UTF-8 string and character libraries

V3 has good language-level Unicode support but only a thin library layer. The architecture is "UTF-8 at the edges, codepoints internally", not "UTF-8 strings everywhere".

9.1 What exists

UCHAR — 32-bit code point.
UString — codepoint-based literals and ARRAY OF UCHAR values.
U'...' / U"..." literals with strict RFC 3629 decoding at compile time (error 234 on malformed input).
UCHR, CHR8, UORD.
UString whole-string assignment to a fixed ARRAY OF UCHAR (count-clamped, zero-codepoint terminated).
+ concatenation for UString literals and single-codepoint operands, plus LEN/LENGTH/HIGH for literals.
runtime/syslib/Utf8 — scalar codec: Decode (one sequence from a byte array to UCHAR and next) and Encode (one UCHAR to bytes). MaxUTF8Length = 4, MaxCodePoint = 1114111.
lib/iso/TextIO — UTF-8 output: UWriteChar, UWriteString, UWriteLn.
Tests: utf8_prog, utext_prog, t_ustrcat, t_ustrassign, t_unicode, plus malformed-literal rejection tests.

Utf8 lives in runtime/syslib and is always shared. TextIO is in lib/iso, so its U* output helpers are ISO-dialect library API; other dialects can import Utf8 manually.

9.2 What is missing

Area	Current state
UTF-8 codec	One codepoint at a time only
Whole-string validation	Manual `Decode` loop only
UTF-8 input	None
UTF-8 output	`TextIO.U*` only
UTF-8 length/index/slice	None
UString library	None
UString comparison/search	None
Dynamic UString	None
Unicode case conversion	None
Unicode character classes	`CharClass` is ASCII-only
Normalization / collation / grapheme clusters	None
Locale/bidi support	None

lib/common/Strings is byte-oriented:

Length counts bytes, not codepoints;
Slice, Pos, Insert, Replace and Delete use byte positions;
Capitalize is explicitly ASCII;
slicing UTF-8 bytes can split a multibyte sequence.

lib/common/CharClass is ASCII-only; there is no UCHAR version and no Unicode category lookup.

Input is byte/line oriented. SysIO, STextIO, IOChan, Files and StreamFile have no UTF-8 decoding or validation policy.

9.3 UString is not a complete string type

A U"..." literal is a descriptor { count, codepoints... }. A declared ARRAY OF UCHAR is a fixed-length array. As a result:

literals have a logical count;
fixed arrays have a capacity plus a zero terminator after assignment;
LEN/HIGH on a fixed array refer to declared bounds, not the number of codepoints actually assigned;
TextIO.UWriteString writes 0 .. HIGH(u), not just through a zero terminator;
there is no general UString equality, comparison, slicing or search.

UString concatenation is bounded and transient. The shim uses a static 4096-codepoint buffer, silently truncates beyond that, and the result is valid only until the next m2ustrcat call. Single-codepoint conversion uses a small rotating pool (16 descriptors). This is adequate for literal expressions, not for a general string runtime.

9.4 Correctness item

Utf8.Encode documents that it returns 0 for a non-scalar value, but its implementation does not reject the surrogate range D800..DFFF. Those values are currently encoded as three-byte UTF-8 sequences, which is invalid UTF-8. Add the surrogate check and a regression test.

9.5 Recommended library design

Keep byte strings and codepoint strings as separate layers. Do not make Strings Unicode-aware in place; existing code expects bytes.

1. UTF-8 byte-string module

For interoperability with files, sockets, C APIs and existing V3 ARRAY OF CHAR strings:

Validate
Length (codepoint count)
ByteLength
Index (codepoint index to byte offset)
Next / Prev
Slice
Compare
Find
ToUString
FromUString

This is the most practical "UTF-8 string" library for V3 because CHAR arrays are already the byte-oriented representation.

2. UString codepoint module

For operations over ARRAY OF UCHAR:

logical length
copy/concat with explicit capacity
compare
slice
find
conversion to/from UTF-8
codepoint iteration

First resolve the length model: either consistently use a zero terminator, or introduce an explicit descriptor with separate length and capacity. Do not mix both.

3. Input/output

Add UTF-8 input alongside output:

whole-line UTF-8 reading;
a documented invalid-input policy;
UReadChar / UReadString (or equivalent);
conversion into UCHAR/UString values.

4. Unicode properties and case

This is a later layer:

case mapping;
character categories;
whitespace;
normalization;
collation.

These need generated Unicode tables, not hand-written ranges.

9.6 Suggested order within the Unicode work

Fix Utf8.Encode surrogate rejection and add a test.
Add the UTF-8 byte-string module (validation, iteration, length/index, conversion).
Add the UString codepoint module and settle its length model.
Add UTF-8 input.
Add Unicode case/property tables if needed.
Keep Strings/CharClass byte/ASCII semantics and add separate Unicode modules rather than changing existing behaviour.

10. Collections library and the TopSpeed B-tree Toolkit

10.1 Current state

V3 has no collections library. What exists is only infrastructure:

lib/common/Storage — ALLOCATE, DEALLOCATE, REALLOCATE
lib/gm2/SysStorage — raw allocation
lib/gm2/DynamicStrings — heap strings
ISO generics: type/value/procedure parameters and kind constraints
generic container demonstrations in tests:
- compiler/tests/showcase23lib.def — DList(T)
- compiler/tests/showcase24stack.def — GStack(T)

Those test modules are proof of concept, not production libraries: they have no Done/Clear, no ownership policy, and use one NEW per node. The MyWork/Strings/DataStructures FIFO/LIFO/dynamic-array files are incomplete and are not part of V3.

10.2 What the TopSpeed B-tree Toolkit is

The real toolkit is installed with the TS-V3 tree. Useful references:

$HOME/bin/Dos/M2/TS-V3/INCLUDE/BTREE.DEF
$HOME/bin/Dos/M2/TS-V3/SRC/BTREE.MOD
$HOME/bin/Dos/M2/TS-V3/DOC/BTREE.DOC
$HOME/bin/Dos/M2/TS-V3/Manuals/B-Tree toolkit.odt

It is a disk-based B-tree/ISAM toolkit, not an in-memory collections library. Its architecture is:

one physical file (FHandle)
multiple logical data and index files (IHandle) in that physical file
fixed or variable-length data records, up to 65,535 bytes
index keys up to 128 bytes
multiple indexes per data file, updated automatically by Add/Delete
application-supplied CompareFunction and KeyFunction
duplicate keys configurable per index
Find/Search (exact and greater-or-equal), Next/Prev/Reset cursors
Add, Delete, Change, Open, OpenData, OpenIndex, Close, Flush, LastError, LastFError
1024-byte pages with a 32-page LRU cache
optional data compression through Pack/APACK2
shared-file locking and multi-tasking support
4 GiB physical-file limit
free-list allocation; the file never shrinks automatically
no automatic compaction
default error handler prints and terminates the program

The current definition also includes AccessMode values such as FixSize, Size16, Size32, NoDealoc and Compress.

The source is 16-bit DOS/OS/2 code: inline x86 assembly, segment arithmetic, TopSpeed pragmas, FIOx, CoreMem, and raw ADDRESS/ARRAY OF BYTE interfaces. It is a design reference, not a portable V3 implementation.

10.3 Good ideas to borrow

Separate logical data and index files, with multiple indexes per data file.
Application-supplied key derivation and comparison.
Cursor-oriented API (Find, Search, Next, Prev, Reset) rather than exposing pages.
Per-handle error state, with an optional handler.
Bounded page cache with LRU replacement.
Configurable duplicate-key handling.
Explicit open/close/flush lifecycle.
A versioned file header and free-space management.
Optional compression as a separate layer.

These map naturally to V3 generics:

Compare : PROCEDURE(a, b : K) : INTEGER;
Hash    : PROCEDURE(k : K) : CARDINAL;
Equal   : PROCEDURE(a, b : K) : BOOLEAN;

10.4 What not to copy

Fatal-error-by-default: a library must return status.
Raw ADDRESS/ARRAY OF BYTE public interfaces: provide typed wrappers.
Fixed 128-byte keys, 1 KiB pages and 4 GiB files: make limits explicit/configurable where practical.
16-bit segment/assembly code.
DOS SHARE/FIOx locking: use V3 file APIs and POSIX locking later.
No compaction: provide a copy/rebuild tool or document the limitation.
Raw binary records without schema/version metadata.
Automatic indexing only for future additions; rebuilding an index must be an explicit operation.
Copying the source code: it is old, platform-specific and carries its own licensing terms.

10.5 Recommended library layout

Keep in-memory collections and the persistent B-tree separate.

In-memory containers

Vector(T)       dynamic array
Deque(T)        FIFO / double-ended queue
Stack(T)        LIFO
HashMap(K,V)    hash table with Hash/Equal callbacks
BTreeMap(K,V)   optional ordered map/set with Compare

Vector is the foundation. Stack and Deque should reuse it or a ring-buffer implementation. Avoid one allocation per element unless stable references are required.

HashMap should use open addressing with tombstones or separate chaining, a load factor around 0.7, and explicit resize/rehash. Add a StringMap(V) convenience wrapper with owned string keys.

BTreeMap is the in-memory ordered structure: comparator callback, first/last/next/prev, range scan, optional duplicate keys, no pager.

Persistent B-tree

BTreeFile  — disk-based index, toolkit-inspired

This is a separate library, not part of Vector/HashMap. It needs:

file format and versioning
page size and cache policy
split/merge and free-space management
cursor state
locking policy
crash/durability policy
compaction/rebuild
optional compression

The TopSpeed toolkit is the checklist for this stage.

10.6 V3 constraints

No GC: containers need explicit Init/Done/Clear.
No automatic deep copy: pointer-element ownership must be documented.
Storage.REALLOCATE exists, but arbitrary pointer arithmetic does not. TopSpeed uses AddAddr/IncAddr/DecAddr assembly helpers; V3 cannot rely on that.
A truly growable Vector(T) needs one of:
- a fixed maximum capacity parameter,
- chunked storage,
- or a small compiler/runtime extension for variable-sized typed arrays.
Generic instances are supported, but multiple actual-type instances should be regression-tested.
Keep array length and capacity separate.
Use records and modules, not classes, for containers.
Do not use a fatal error handler as the default.

Suggested error model:

TYPE Status = (Ok, Empty, NotFound, InvalidIndex,
               OutOfMemory, DuplicateKey);

If enum handling is awkward in a given context, use integer status codes or BOOLEAN plus LastError.

10.7 Suggested order within the collections work

Vector(T)
Deque(T) / Stack(T)
HashMap(K,V)
StringMap(V)
In-memory BTreeMap(K,V)
Persistent BTreeFile based on the TopSpeed architecture

The TopSpeed B-tree Toolkit should not delay steps 1–3.

11. Suggested order of future work

TBITSIZE (small, standard-facing).
ADDADR/SUBADR/DIFADR (small, ISO SYSTEM).
Set ordering (bounded gm2 compatibility; fixes largeset5).
Complex types: choose ISO built-ins versus a nonstandard value-record library; remove the misleading COMPLEX* aliases.
Pragmas: native <* ... *> parser/IR plus a low-risk semantic subset; later add the TopSpeed adapter and ABI/layout pragmas.
UTF-8 string/character libraries: fix Utf8.Encode, add byte-string and UString modules, then UTF-8 input.
Collections: Vector, Deque/Stack, HashMap, then optional BTreeMap; persistent BTreeFile last.
ISO COROUTINES subset, then interrupt/protection layer.
Dynamic SET CONST and deep composite copies.
Parser error recovery and diagnostics.

External assembly needs no compiler work; document it as the supported low-level escape hatch.

future-work-2026-10-25.md 35 KB Cronologia Originale