Theia-Modula-2/grammars/FreePascal

FreePascal validator pilot

free_pascal.atg describes Free Pascal / Delphi-dialect Object Pascal. It needed a full repair pass to build with Coco/R V1.53 (CR) and GNU Modula-2 (gm2 -fiso), all diagnosed against fpc -Mdelphi as oracle (default mode lacks classes; -Miso lacks units, strings and most extensions):

1. Bare ; is not a Coco/R literal — every separator is ";".
2. Unterminated productions get their . (Statement ended with a bare ;).
3. "" is illegal — empty positions use tolerant list shapes ([ X { ";" [ X ] } ], progress-guaranteed so the parser cannot hang).
4. ident cannot be both token and production (literals renamed to pnumber/pstring); junk EndProgram/EOF lines removed; the start production matches COMPILER.
5. Left-recursive Primary "^" rewritten as postfix.
6. Undefined nonterminals filled in (LabelDecl, PointerType, ParDeclList→FormalParams, Stmts→Statements, noQuote1, MethodSig, TypeName-equivalents, UsesClause).
7. Name collisions in generated CASE statements fixed by merging: duplicate for..to/downto, duplicate Type/StructType and ArrayType/ParamType entries, ident-led assignment-vs-call (single IdentTail dispatch), [class]-prefixed routine twins.
8. Grammar gaps closed to oracle level: TypeIdent (named types), Compound program bodies, trailing-;-tolerant sections/fields/ case-items, {}////{$} comments (with eol = CHR(10) — CHR(13) made // eat to EOF), and/or/xor/shl/shr/sar words, bare parameterless calls, numeric statement labels + goto (+LabelSection), enum labels, formal parameter groups (with var/out/const, defaults), subrange/expression array bounds, const expressions, CASE branches take a single statement, multi-routine blocks, preal floats, $/&/% literals, #nn char chains, enums, dotted names, units (top-level, with interface/implementation/init/final + repeated uses-groups), library headers, classes (sealed/abstract/parents/helpers, visibility, fields, generic/static methods, properties with positional accessors) and interfaces, class/static routine implementations with owner dots, generics (generic prefix, <T> params, specialize, <> args), operator declarations, procedural/of-object types, try/except/on/finally, raise, exit/break, inherited, is/as, in, for..in, threadvar, resourcestring, absolute, forward/abstract/ external (all arities) + calling-convention/method directives as a closed vocabulary, on-clauses by position (var on stays legal), case..else, shortstrings, packed/bitpacked, set literals with ranges, deref/call-then-select chains, nil.
9. Contextual words probed against the oracle: static, read, write, message, assembler, nostackframe, register, reference, object, weak, on, Supports stay identifiers; overload/varargs/external/set/threadvar/operator-words are legal dotted unit-name parts.
10. Corpus-driven round (fpcsrc): bare-class forwards, legacy object types, objcclass with protocol parents, class of references, class var blocks, class/static operators (Implicit/Explicit, owner dots), ** power, +=-family assignments, @ address-of, inherited calls in expressions, write-width : params, external lib/name chains, GUID'd interfaces, const-indexed properties, array of const, array[N]/(N) string lengths, shortstring/enum/dotted case labels, case..of in records (variants), nested const/type/var blocks in records/classes, visibility in records, record methods, empty case branches, juxtaposed record fields (directive tails eat ;), interface/implementation-level resourcestring/threadvar, optional implementation, comment-only fragments (driver skips FPC on immediate EOF), BOM bytes ignored, X_PACKED packing macro (2 files, documented), XIdent (out/static/register/far/near stay usable as declaration names, statements and expressions — probed: var /const as names are rejected by the oracle too), public as unit-var directive vs class visibility (split VarDecl/ VarDeclNoTail), far/near/static/assembler/ nostackframe/syscall/extdecl routine directives, constrained generic params (<E: class>), >=-tolerant generic closers (GClose), generic parents (class(specialize TBase<Integer>)), string-adjacent #nn/^M chains, paren const-lists with :/;/ trailing-;, enum =/:= values, array[seg:ofs], initialized var x: T = V, absolute (expr), builtin-type casts (integer(x)), deref-after-parens, of object procedural types, cdecl-family type tails, syscall in interfaces, empty then/else/do bodies, variant case <type> of via Type [":" Type], indexed property ...[const i: T], ...; deprecated; property tails, X_PACKED packing macro (2 files).
11. Toolchain limits discovered: the grammar is at Coco/R V1.53's reliable capacity — additions past a point silently degrade code generation with no diagnostic beyond benign LL(1) notes. Two failure modes seen: IF FALSE/WHILE FALSE conditions (a {...} loop over ~10+ alternatives, or any ANY-complement token), and silently pruned dispatch branches (even a minimal "asm" "end" block broke array indexing with zero warnings). All pilot build.sh scripts fail the build on the FALSE pattern, and every rebuild is followed by the differential battery + tests/deep.pas (the latter caught the pruning). Consequence: asm bodies of any kind stay a documented gap. {$define} macro expansion in general remains a documented gap, as do conditional-define branch selection ({$ifc} units parse all branches literally) and include-only test fragments the oracle itself rejects.

The generated parser gets a strict EOF check (IF sym # 0 THEN SynError(0) after the start symbol in Parse, patched in by build.sh with a loud assertion): without it, the Coco/R driver never scans past the program's final ., silently ignoring trailing garbage — and worse, masking any failure that leaves a . behind. This makes the validator marginally stricter than fpc, which accepts trailing junk after END. — degenerate input only, documented here.

if/while/for/with/on-do branches take a single Statement (standard Pascal — plural branches greedily swallow the ; separating case items, so if..then begin..end; inside a branch broke); repeat/try/begin bodies stay plural (terminator-closed). Declaration sections are order-free (DeclPart is a loop, since real units interleave const after routines).

Known gaps (all documented, corpus-measured): asm bodies of any kind (toolchain capacity — even an empty-block production silently broke unrelated dispatches); message N with numeric args; {$define} macro expansion (X_PACKED packing prefix is accepted as a literal — 2 files); conditional-define branches are parsed literally, so multi-branch {$ifc} units can mismatch the oracle's selected branch.

Build

./build.sh        # -> build/FPC (build/ is git-ignored)

Needs CR (Coco/R), gm2 -fiso, the parser.frm/scanner.frm frames and FileIO.def/.mod — all resolved from the environment with sensible defaults (CRFRAMES, M2LIB, CR, GM2).

Differential results (FPC vs fpc -Mdelphi)

Accept battery + tests/deep.pas + tests/class.pas: 0 false positives. Broken mutants: 0 missed, positions close. Full fpcsrc corpus (11,391 files, per-file {$mode} oracle, fail-tests excluded): 7,613 clean-agree, 2,287 validator-FP, 232 validator-miss (nearly all headerless include-fragments the oracle rejects standalone — documented leniency), 567 oracle-parse-reject. FP is dominated by the documented gaps: asm bodies (~740 files) and conditional-define branches parsed literally.