Add proof test for JSON tokenizer minus state EOF handling - #1549
Conversation
Through the public tokenize() API, parseMinusState never sees input===null: the underlying JS tokenizer always emits its own eof token first, which resets state to 'def' before the sentinel null is seen. Call the private state handler directly to exercise the branch anyway. Coverage (npm run cov): branches 98.598% -> 98.619% (4640/4706 -> 4643/4708), lines and functions unchanged at 99.955% / 99.803%. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBPH2uUAyP4GXcZwJ7EZkX
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
functionalscript | fe928fa | Commit Preview URL Branch Preview URL |
Aug 14 2026, 01:40 PM |
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBPH2uUAyP4GXcZwJ7EZkX # Conflicts: # fjs/media/json/tokenizer/module.f.mjs
o2alexanderfedin
left a comment
There was a problem hiding this comment.
Reviewed at fe928fa2, baseline origin/main = 6d26e264. Proof-only, one file. Approving — the case has real, two-sided-verified kill power, and it avoids the two weaknesses that made earlier cases in this family vacuous.
Two-sided mutation test
Three independent mutations of the target branch (parseMinusState's input === null arm), each applied with node -e and grep-confirmed to have landed:
| mutation | at fe928fa2 |
on origin/main |
|---|---|---|
state { kind: 'def' } → { kind: '-' } |
fail 1 / 2652 pass | 2652 pass, 0 fail |
tokens [{ kind: 'error', … }] → empty |
fail 1 / 2652 pass | 2652 pass, 0 fail |
token kind 'error' → 'null' |
fail 1 / 2652 pass | 2652 pass, 0 fail |
In every case the single failure is parseMinusStateEof and nothing else, so the case is the sole cause and nothing on main covers this branch. npm test at the head is 2653 pass / 0 fail against 2652 on main — exactly the one new case.
Against the known weaknesses in this family
- #1518 (pinned the error message, so a branch change preserving the message survived): not repeated. This case pins
kind, notmessage, so it keys on the branch's semantic output rather than its prose. - #1525 / #1528 (pinned one field, so a mutant with the right tag and a clobbered state survived): not repeated. It pins both halves of the returned pair —
state.kindand the token list's length and kind. Thedef→-mutation above is exactly the #1525 mutant, and it dies here. - #1535 was correctly not a defect because
proof.throwasserts only that something throws; that reasoning does not apply here — these areassertEqon concrete values, so the distinction is moot.
Coverage delta (real numbers, npm run cov)
npm run cov produced a usable report in this environment on this run (it has reported a vacuous 100.00 over 0 tests before; it did not here — 2653 cases ran).
fjs/media/json/tokenizer/module.f.mjs branch 93.75 -> 97.06
all files branch 98.60 -> 98.62
Line and function coverage were already 100.00 in that file and are unchanged; the gain is entirely the branch this case targets.
The unreachability claim in the comment
The comment asserts the branch cannot be reached through the public tokenize. I checked rather than took it: I replaced the branch body with a throw and ran tokenize over -, - , - , --, -x, -1, -\n, -, [-, [1,-], {"a":-, and "". None hit it — the JS tokenizer's own eof token takes the default arm and resets the state to def before the appended null sentinel arrives. The comment is accurate.
That does mean the branch is currently dead through the public API, and one could argue for deleting it instead of covering it. I do not think that is the right call here: parseMinusState is a StateScan handler whose input type includes null, so the arm is part of a total function over its declared domain, and covering it is cheaper and safer than narrowing the type. Noting it only so the choice is explicit.
Conventions
export const proofinmodule.f.mjs(rather thanproof.f.mjs) is required here becauseparseMinusStateis module-private, and it is established practice — 10module.f.mjsfiles already do it, including the siblingfjs/media/json/parser/module.f.mjsandfjs/js/tokenizer/module.f.mjs.- No CHANGELOG entry, correctly: §8.3 exempts proof-only changes, and the diff touches nothing else.
npx tsc --noEmitexit 0.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Summary
This PR adds a proof test to verify the JSON tokenizer correctly handles the edge case where input ends immediately after a lone minus sign (
-).Key Changes
toArrayutility from the list module to support the new testassertEqassertion helper for test validationproofexport object containingparseMinusStateEoftest case that:parseMinusStatehandler withnull(EOF sentinel)'def'(default)Implementation Details
The test directly calls the internal
parseMinusStatefunction with the EOF sentinel to exercise a code path that cannot be reached through normaltokenizeusage. The underlying JS tokenizer always emits its own EOF token first, which resets the state before the sentinelnullis encountered. This proof test ensures the handler gracefully produces an error token in this edge case.https://claude.ai/code/session_01FBPH2uUAyP4GXcZwJ7EZkX