Просмотр исходного кода

Compiler: a string literal was eating the rest of the source

Reported as "error 102 at relative pos 48" on a five-line hello-world,
with the editor landing past the final '.' of the program.  The cause was
not the error but the string scanner: every reported position for a
multi-character literal was exactly Length(), i.e. one past the last
character of the buffer, so the editor had nowhere valid to put the
cursor.

RdConst's multi-character branch read:

    WHILE (ORD (CurCh ()) # q) AND (CurCh () # 0C) DO
       IF ORD (PeekAhead (1)) = q THEN
          DropCh (GetCh ()) ; DropCh (GetCh ())
       ELSE
          DropCh (GetCh ())
       END
    END

The condition tests the current character, but the "closing quote
detected" branch consumes two characters: the last content character
AND the quote.  The cursor is now past the quote, so the next condition
test inspects the character *after* the literal, finds it is not a
quote, and the scan runs on to end-of-buffer.  The only exit is 0C.  So
'hi' consumed the ')', the ';', 'end' and '.', and the reported position
was the end of the file.  The IF at 1155 that was meant to consume the
closing quote was unreachable for any string of two or more characters.

Worse than a bad caret: it destroyed the parse.  Anything after a
multi-character literal was consumed as string contents, so a genuine
later error was misattributed to end-of-file, and the trailing
MatchDelim('.') failed as collateral.  Only strings of exactly one
character escaped, which is why writeln('a') always worked and
writeln('hi') never did.

Fixed by stopping on the quote itself and treating a doubled quote as
one embedded quote character, so Pascal's 'don''t' still works.
Unterminated literals now report EUnknown instead of running away.

    t02_writeln   102 at pos 33 -> 33   (was past end of buffer)
    t17_two_str   102 at pos 34 -> 34   (was 46, the buffer length)
    toto.pas      102 at pos 38 -> 38   (was 48, the buffer length)

The five remaining ERR fixtures are still unimplemented features, not
parser bugs: four need the string runtime, t14 needs 'array' at its
point of use, uierror is the deliberate syntax error.

The compile matrix also stops being a number a human counted by hand.
tests/fixtures/expected.tsv pins the verdict AND the numbers per
fixture and run_compile_tests.sh compares them, so a wrong error
position now fails the suite instead of needing a squint.  Verified the
suite is not vacuous: with the fix reverted it reports 19/23 and exits
1, with the fix it reports 23/23 and exits 0.

Unchanged elsewhere: tpshell 120792 bytes, uitest 10/10.
Eric Streit 2 недель назад
Родитель
Сommit
42402a0117
3 измененных файлов с 124 добавлено и 4 удалено
  1. 20 3
      shell/Compiler.mod
  2. 45 0
      shell/tests/fixtures/expected.tsv
  3. 59 1
      shell/tests/run_compile_tests.sh

+ 20 - 3
shell/Compiler.mod

@@ -1145,9 +1145,26 @@ BEGIN
          ELSE
             isStr := TRUE ;
             cls := TString ;
-            WHILE (ORD (CurCh ()) # q) AND (CurCh () # 0C) DO
-               IF ORD (PeekAhead (1)) = q THEN
-                  DropCh (GetCh ()) ; DropCh (GetCh ())
+            (* Scan to the closing quote.  NB: the loop condition must test
+               the CURRENT character and only the current character.  The
+               obvious-looking "while CurCh # quote" with a
+               "if PeekAhead(1) = quote then consume two" body is wrong:
+               consuming the quote moves the cursor past it, so the next
+               condition test sees the character AFTER the literal, is
+               satisfied, and the scan runs on to end-of-buffer - which
+               silently eats the rest of the program and makes every later
+               error point at end-of-file.  Stop on the quote itself, and
+               treat a doubled quote as one embedded quote character. *)
+            LOOP
+               IF ORD (CurCh ()) = q THEN
+                  IF ORD (PeekAhead (1)) = q THEN
+                     DropCh (GetCh ()) ; DropCh (GetCh ())   (* '' inside *)
+                  ELSE
+                     EXIT                                (* closing quote *)
+                  END
+               ELSIF (CurCh () = 0C) OR (ORD (CurCh ()) = 0DH) THEN
+                  Err (EUnknown) ;                         (* unterminated *)
+                  EXIT
                ELSE
                   DropCh (GetCh ())
                END

+ 45 - 0
shell/tests/fixtures/expected.tsv

@@ -0,0 +1,45 @@
+# Expected results for tests/fixtures/*.pas  -  one line per fixture:
+#
+#   <fixture-basename>	OK	<code>	<data>
+#   <fixture-basename>	ERR	<tp3-error-no>	<relative-pos>
+#
+# Both the verdict AND the numbers are asserted by run_compile_tests.sh, so a
+# regression in the *reported error position* fails the suite just like a
+# regression in the generated code.  That matters: error positions were once
+# silently wrong (a multi-character string literal swallowed the rest of the
+# source, so every later error pointed at end-of-file), and nothing caught it
+# because the old harness only printed the position for a human to squint at.
+#
+# Re-baseline deliberately, never to make a red test go green: if a fixture's
+# expectation has to change because the compiler legitimately improved, change
+# it in the same commit as the fix and say why in the message.
+#
+# ERR 102 = ENoLib, the original's "not implemented" path.  The five ERR rows
+# below are all unimplemented features, not parser bugs:
+#   t02 t03 t05 t17  multi-character string literal (string runtime pending)
+#   t14               'array [..] of <type>' at its point of use
+#   uierror           deliberate syntax error, pinned by the editor UI test
+
+t01_minimal	OK	26	260
+t02_writeln	ERR	102	33
+t03_inline_comment	ERR	102	41
+t04_var	OK	45	262
+t05_own_line_comment	ERR	102	65
+t06_two_args	OK	49	260
+t07_big	OK	99	262
+t08_const	OK	32	262
+t09_if	OK	64	262
+t10_while	OK	72	262
+t11_for	OK	66	262
+t12_repeat	OK	69	262
+t13_proc	OK	50	262
+t14_types	ERR	102	83
+t15_label	OK	35	262
+t16_str1	OK	39	260
+t17_two_str	ERR	102	34
+t18_writeln_bare	OK	29	260
+t19_int1	OK	39	260
+t20_str3	OK	59	260
+t21_mixed	OK	59	260
+t22_case	OK	93	262
+uierror	ERR	41	331

+ 59 - 1
shell/tests/run_compile_tests.sh

@@ -43,4 +43,62 @@ fi
 echo "compiletest built (p1_rc=$p1, phase 1 rc=1 is the expected rollup)"
 
 FX=${1:-tests/fixtures}
-ls "$FX"/*.pas 2>/dev/null | ./compiletest
+
+# ---- assert against tests/fixtures/expected.tsv ---------------------------
+# Previously this script only printed a verdict, so "17 of 23" was a number a
+# human counted by hand and a wrong error POSITION was invisible.  Both the
+# verdict and the numbers are compared now, and the exit status is meaningful.
+EXP="$FX/expected.tsv"
+if [ ! -f "$EXP" ]; then
+    echo "no expectations file $EXP - printing only"
+    ls "$FX"/*.pas 2>/dev/null | ./compiletest
+    exit 0
+fi
+
+RAW=$(mktemp) || exit 9
+ls "$FX"/*.pas 2>/dev/null | ./compiletest > "$RAW"
+
+# expected row for a fixture basename, normalised to single spaces
+# (only rows that actually carry a verdict count, so the wrapped comment
+#  lines in expected.tsv can never be mistaken for data)
+expect_for() {
+    awk -F'\t' -v n="$1" '
+        ($2 == "OK" || $2 == "ERR") && $1 == n {
+            out = ""
+            for (i = 2; i <= NF; i++)
+                out = (out == "") ? $i : out " " $i
+            print out
+            exit
+        }
+    ' "$EXP"
+}
+
+pass=0; fail=0
+while IFS= read -r base; do
+    actual=$(grep -F "$base" "$RAW" \
+             | head -1 \
+             | tr -d '\r' \
+             | sed -E 's/^[^ ]+[[:space:]]+//' \
+             | tr -s ' ' \
+             | sed -E 's/^OK code=([0-9]+) data=([0-9]+)$/OK \1 \2/;
+                       s/^ERROR ([0-9]+) at pos ([0-9]+)$/ERR \1 \2/')
+    want=$(expect_for "$base" | tr -s ' ')
+    if [ -z "$want" ]; then
+        printf '  %-24s NO_EXPECTATION  got: %s\n' "$base" "$actual"
+        fail=$((fail + 1))
+    elif [ "$actual" = "$want" ]; then
+        printf '  %-24s PASS  %s\n' "$base" "$actual"
+        pass=$((pass + 1))
+    else
+        printf '  %-24s FAIL\n      expected: %s\n      actual:   %s\n' \
+               "$base" "$want" "$actual"
+        fail=$((fail + 1))
+    fi
+done <<EOF
+$(ls "$FX"/*.pas 2>/dev/null | xargs -n1 basename | sed 's/\.pas$//')
+EOF
+
+echo "----------------------------------------------------------------"
+echo "compile matrix: $pass passed, $fail failed (of $((pass + fail)))"
+rm -f "$RAW"
+[ "$fail" -eq 0 ] || exit 1