docs: analyse UTF-8 library support

docs/INDEX.md

@@ -35,8 +35,9 @@ bisecting *why* a decision was made, not for reading end-to-end).
 ## Future-work analysis
 
 - **`future-work-2026-10-25.md`** — TBITSIZE, ADDRESS arithmetic, set
-  ordering, complex types, pragmas (TopSpeed vs portable), PIM vs ISO
-  coroutines, external assembly, dynamic-set design and suggested order.
+  ordering, complex types, pragmas (TopSpeed vs portable), UTF-8 string
+  libraries, PIM vs ISO coroutines, external assembly, dynamic-set
+  design and suggested order.
 
 ## Designs / running records
 

docs/future-work-2026-10-25.md

@@ -2,9 +2,9 @@
 
 This note records the design discussions from 2026-10-25 so the
 questions do not have to be re-derived later.  It covers `TBITSIZE`,
-address arithmetic, set ordering, complex types, pragmas, PIM versus ISO
-coroutines, external assembly, and the dynamic-set design.  It is not a
-commitment to implement everything.
+address arithmetic, set ordering, complex types, pragmas, UTF-8 string
+libraries, PIM versus ISO coroutines, external assembly, and the
+dynamic-set design.  It is not a commitment to implement everything.
 
 **State at the time of writing:** suite 310/310, one-shot fixpoint OK
 (3,845,439 bytes), per-object fixpoint OK (11 objects), corpus
@@ -702,7 +702,167 @@ success.
 
 ---
 
-## 9. Suggested order of future work
+## 9. UTF-8 string and character libraries
+
+V3 has good language-level Unicode support but only a thin library
+layer.  The architecture is "UTF-8 at the edges, codepoints internally",
+not "UTF-8 strings everywhere".
+
+### 9.1 What exists
+
+- `UCHAR` — 32-bit code point.
+- `UString` — codepoint-based literals and `ARRAY OF UCHAR` values.
+- `U'...'` / `U"..."` literals with strict RFC 3629 decoding at
+  compile time (error 234 on malformed input).
+- `UCHR`, `CHR8`, `UORD`.
+- `UString` whole-string assignment to a fixed `ARRAY OF UCHAR`
+  (count-clamped, zero-codepoint terminated).
+- `+` concatenation for UString literals and single-codepoint
+  operands, plus `LEN`/`LENGTH`/`HIGH` for literals.
+- `runtime/syslib/Utf8` — scalar codec:
+  `Decode` (one sequence from a byte array to `UCHAR` and `next`) and
+  `Encode` (one `UCHAR` to bytes).  `MaxUTF8Length = 4`,
+  `MaxCodePoint = 1114111`.
+- `lib/iso/TextIO` — UTF-8 output:
+  `UWriteChar`, `UWriteString`, `UWriteLn`.
+- Tests: `utf8_prog`, `utext_prog`, `t_ustrcat`, `t_ustrassign`,
+  `t_unicode`, plus malformed-literal rejection tests.
+
+`Utf8` lives in `runtime/syslib` and is always shared.  `TextIO` is in
+`lib/iso`, so its `U*` output helpers are ISO-dialect library API; other
+dialects can import `Utf8` manually.
+
+### 9.2 What is missing
+
+| Area | Current state |
+| --- | --- |
+| UTF-8 codec | One codepoint at a time only |
+| Whole-string validation | Manual `Decode` loop only |
+| UTF-8 input | None |
+| UTF-8 output | `TextIO.U*` only |
+| UTF-8 length/index/slice | None |
+| UString library | None |
+| UString comparison/search | None |
+| Dynamic UString | None |
+| Unicode case conversion | None |
+| Unicode character classes | `CharClass` is ASCII-only |
+| Normalization / collation / grapheme clusters | None |
+| Locale/bidi support | None |
+
+`lib/common/Strings` is byte-oriented:
+
+- `Length` counts bytes, not codepoints;
+- `Slice`, `Pos`, `Insert`, `Replace` and `Delete` use byte positions;
+- `Capitalize` is explicitly ASCII;
+- slicing UTF-8 bytes can split a multibyte sequence.
+
+`lib/common/CharClass` is ASCII-only; there is no `UCHAR` version and
+no Unicode category lookup.
+
+Input is byte/line oriented.  `SysIO`, `STextIO`, `IOChan`, `Files` and
+`StreamFile` have no UTF-8 decoding or validation policy.
+
+### 9.3 UString is not a complete string type
+
+A `U"..."` literal is a descriptor `{ count, codepoints... }`.  A
+declared `ARRAY OF UCHAR` is a fixed-length array.  As a result:
+
+- literals have a logical count;
+- fixed arrays have a capacity plus a zero terminator after assignment;
+- `LEN`/`HIGH` on a fixed array refer to declared bounds, not the number
+  of codepoints actually assigned;
+- `TextIO.UWriteString` writes `0 .. HIGH(u)`, not just through a zero
+  terminator;
+- there is no general UString equality, comparison, slicing or search.
+
+UString concatenation is bounded and transient.  The shim uses a static
+4096-codepoint buffer, silently truncates beyond that, and the result is
+valid only until the next `m2ustrcat` call.  Single-codepoint conversion
+uses a small rotating pool (16 descriptors).  This is adequate for
+literal expressions, not for a general string runtime.
+
+### 9.4 Correctness item
+
+`Utf8.Encode` documents that it returns 0 for a non-scalar value, but
+its implementation does not reject the surrogate range `D800..DFFF`.
+Those values are currently encoded as three-byte UTF-8 sequences, which
+is invalid UTF-8.  Add the surrogate check and a regression test.
+
+### 9.5 Recommended library design
+
+Keep byte strings and codepoint strings as separate layers.  Do not make
+`Strings` Unicode-aware in place; existing code expects bytes.
+
+**1. UTF-8 byte-string module**
+
+For interoperability with files, sockets, C APIs and existing V3
+`ARRAY OF CHAR` strings:
+
+- `Validate`
+- `Length` (codepoint count)
+- `ByteLength`
+- `Index` (codepoint index to byte offset)
+- `Next` / `Prev`
+- `Slice`
+- `Compare`
+- `Find`
+- `ToUString`
+- `FromUString`
+
+This is the most practical "UTF-8 string" library for V3 because `CHAR`
+arrays are already the byte-oriented representation.
+
+**2. UString codepoint module**
+
+For operations over `ARRAY OF UCHAR`:
+
+- logical length
+- copy/concat with explicit capacity
+- compare
+- slice
+- find
+- conversion to/from UTF-8
+- codepoint iteration
+
+First resolve the length model: either consistently use a zero
+terminator, or introduce an explicit descriptor with separate length and
+capacity.  Do not mix both.
+
+**3. Input/output**
+
+Add UTF-8 input alongside output:
+
+- whole-line UTF-8 reading;
+- a documented invalid-input policy;
+- `UReadChar` / `UReadString` (or equivalent);
+- conversion into `UCHAR`/UString values.
+
+**4. Unicode properties and case**
+
+This is a later layer:
+
+- case mapping;
+- character categories;
+- whitespace;
+- normalization;
+- collation.
+
+These need generated Unicode tables, not hand-written ranges.
+
+### 9.6 Suggested order within the Unicode work
+
+1. Fix `Utf8.Encode` surrogate rejection and add a test.
+2. Add the UTF-8 byte-string module (validation, iteration,
+   length/index, conversion).
+3. Add the UString codepoint module and settle its length model.
+4. Add UTF-8 input.
+5. Add Unicode case/property tables if needed.
+6. Keep `Strings`/`CharClass` byte/ASCII semantics and add separate
+   Unicode modules rather than changing existing behaviour.
+
+---
+
+## 10. Suggested order of future work
 
 1. `TBITSIZE` (small, standard-facing).
 2. `ADDADR`/`SUBADR`/`DIFADR` (small, ISO SYSTEM).
@@ -711,9 +871,11 @@ success.
    library; remove the misleading `COMPLEX*` aliases.
 5. Pragmas: native `<* ... *>` parser/IR plus a low-risk semantic
    subset; later add the TopSpeed adapter and ABI/layout pragmas.
-6. ISO `COROUTINES` subset, then interrupt/protection layer.
-7. Dynamic `SET CONST` and deep composite copies.
-8. Parser error recovery and diagnostics.
+6. UTF-8 string/character libraries: fix `Utf8.Encode`, add byte-string
+   and UString modules, then UTF-8 input.
+7. ISO `COROUTINES` subset, then interrupt/protection layer.
+8. Dynamic `SET CONST` and deep composite copies.
+9. Parser error recovery and diagnostics.
 
 External assembly needs no compiler work; document it as the supported
 low-level escape hatch.