summary_unicode.md 2.4 KB

Step: Unicode — increment 1 (UCHAR + U-literals)

Tags v3-unicode-1. Suite 118/118; self-hosting fixpoint still OK (bootstrap/fixpoint.sh, image 2,060,379 bytes).

Landed

  • UCHAR — 32-bit codepoint (0..1114111), QBE w, distinct from CHAR/INTEGER (class ClUChar, form FUChar). Usable in VAR/array declarations and designators (ARRAY OF UCHAR).
  • U'..' / U".." literals — Coco/R ustring token; the parser action strict-RFC3629-decodes the bytes between the quotes (QbeGen.DeclUStr). One codepoint → UCHAR immediate; otherwise a UString descriptor data $ustrN = { l count, w cp0, … } (QbeGen.FlushUStrings, emitted at EndModule).
  • Strict decoding — rejects overlongs, surrogates (D800..DFFF), >10FFFF, and truncated sequences → semantic error 234 invalid UTF-8 in U-literal (new message in compiler.frm).
  • Conversions — UCHR(c:CHAR):UCHAR (widening, identity); CHR8(u:UCHAR):CHAR with an l-domain 0..255 range check (traps via $abort, rc 134); UORD(u):LONGCARD.
  • Comparisons — OrdCheck/EqCheck accept UCHAR vs UCHAR, and (generally useful) LONGCARD vs INTEGER; the existing long-family comparison path widens the int side.

Tests (compiler/run_tests.sh)

  • t_unicode.mod (exit 31): U'é', U'😀', U'€', UCHR, CHR8, UORD.
  • t_trap_chr8.mod: CHR8(U'😀') → abort (rc 134).
  • t_bad_overlong.mod, t_bad_surrogate.mod, t_bad_utf8big.mod, t_bad_truncutf.mod: expect invalid UTF-8 in U-literal (234). These carry raw invalid bytes; .LST output is git-ignored.

Deferred to increment 2

  • UString backing/assignment (array-of-UCHAR view, whole-string assignment, index/HIGH).
  • runtime/syslib/Utf8 (Decode/Encode, MaxUTF8Length=4, MaxCodePoint=1114111), shared by lexer and output libs.
  • TextIO.UWriteChar/UWriteString/UWriteLn (UTF-8 output via SysIO); raw WriteChar stays byte-oriented.
  • Suppressing .ssa on error 234 (the image is still written).

Files

compiler/src/{SymTab.def,SymTab.mod} (UCHAR type, ordinal/equality rules), compiler/src/{QbeGen.def,QbeGen.mod} (DeclUStr/FlushUStrings, pool + descriptor emission), compiler/src/M2.atg (ustring token + Fact branch + UCHR/CHR8/UORD), compiler/src/compiler.frm (message 234), compiler/run_tests.sh, compiler/tests/t_unicode.mod, compiler/tests/t_trap_chr8.mod, compiler/tests/t_bad_*.mod.