Skip to content

Add proof test for JSON tokenizer minus state EOF handling - #1549

Merged
sergey-shandar merged 3 commits into
mainfrom
claude/zen-archimedes-5s54pp
Aug 14, 2026
Merged

Add proof test for JSON tokenizer minus state EOF handling#1549
sergey-shandar merged 3 commits into
mainfrom
claude/zen-archimedes-5s54pp

Conversation

@sergey-shandar

Copy link
Copy Markdown
Contributor

Summary

This PR adds a proof test to verify the JSON tokenizer correctly handles the edge case where input ends immediately after a lone minus sign (-).

Key Changes

  • Import toArray utility from the list module to support the new test
  • Import assertEq assertion helper for test validation
  • Add proof export object containing parseMinusStateEof test case that:
    • Directly invokes the private parseMinusState handler with null (EOF sentinel)
    • Verifies the state resets to 'def' (default)
    • Confirms an error token is produced for the incomplete minus sign
    • Documents why this branch is unreachable through the public API but worth testing

Implementation Details

The test directly calls the internal parseMinusState function with the EOF sentinel to exercise a code path that cannot be reached through normal tokenize usage. The underlying JS tokenizer always emits its own EOF token first, which resets the state before the sentinel null is encountered. This proof test ensures the handler gracefully produces an error token in this edge case.

https://claude.ai/code/session_01FBPH2uUAyP4GXcZwJ7EZkX

Through the public tokenize() API, parseMinusState never sees input===null:
the underlying JS tokenizer always emits its own eof token first, which
resets state to 'def' before the sentinel null is seen. Call the private
state handler directly to exercise the branch anyway.

Coverage (npm run cov): branches 98.598% -> 98.619% (4640/4706 -> 4643/4708),
lines and functions unchanged at 99.955% / 99.803%.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FBPH2uUAyP4GXcZwJ7EZkX
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
functionalscript fe928fa Commit Preview URL

Branch Preview URL
Aug 14 2026, 01:40 PM

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FBPH2uUAyP4GXcZwJ7EZkX

# Conflicts:
#	fjs/media/json/tokenizer/module.f.mjs

@o2alexanderfedin o2alexanderfedin left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at fe928fa2, baseline origin/main = 6d26e264. Proof-only, one file. Approving — the case has real, two-sided-verified kill power, and it avoids the two weaknesses that made earlier cases in this family vacuous.

Two-sided mutation test

Three independent mutations of the target branch (parseMinusState's input === null arm), each applied with node -e and grep-confirmed to have landed:

mutation at fe928fa2 on origin/main
state { kind: 'def' }{ kind: '-' } fail 1 / 2652 pass 2652 pass, 0 fail
tokens [{ kind: 'error', … }]empty fail 1 / 2652 pass 2652 pass, 0 fail
token kind 'error''null' fail 1 / 2652 pass 2652 pass, 0 fail

In every case the single failure is parseMinusStateEof and nothing else, so the case is the sole cause and nothing on main covers this branch. npm test at the head is 2653 pass / 0 fail against 2652 on main — exactly the one new case.

Against the known weaknesses in this family

  • #1518 (pinned the error message, so a branch change preserving the message survived): not repeated. This case pins kind, not message, so it keys on the branch's semantic output rather than its prose.
  • #1525 / #1528 (pinned one field, so a mutant with the right tag and a clobbered state survived): not repeated. It pins both halves of the returned pair — state.kind and the token list's length and kind. The def- mutation above is exactly the #1525 mutant, and it dies here.
  • #1535 was correctly not a defect because proof.throw asserts only that something throws; that reasoning does not apply here — these are assertEq on concrete values, so the distinction is moot.

Coverage delta (real numbers, npm run cov)

npm run cov produced a usable report in this environment on this run (it has reported a vacuous 100.00 over 0 tests before; it did not here — 2653 cases ran).

fjs/media/json/tokenizer/module.f.mjs   branch  93.75 -> 97.06
all files                               branch  98.60 -> 98.62

Line and function coverage were already 100.00 in that file and are unchanged; the gain is entirely the branch this case targets.

The unreachability claim in the comment

The comment asserts the branch cannot be reached through the public tokenize. I checked rather than took it: I replaced the branch body with a throw and ran tokenize over -, - , - , --, -x, -1, -\n, -, [-, [1,-], {"a":-, and "". None hit it — the JS tokenizer's own eof token takes the default arm and resets the state to def before the appended null sentinel arrives. The comment is accurate.

That does mean the branch is currently dead through the public API, and one could argue for deleting it instead of covering it. I do not think that is the right call here: parseMinusState is a StateScan handler whose input type includes null, so the arm is part of a total function over its declared domain, and covering it is cheaper and safer than narrowing the type. Noting it only so the choice is explicit.

Conventions

  • export const proof in module.f.mjs (rather than proof.f.mjs) is required here because parseMinusState is module-private, and it is established practice — 10 module.f.mjs files already do it, including the sibling fjs/media/json/parser/module.f.mjs and fjs/js/tokenizer/module.f.mjs.
  • No CHANGELOG entry, correctly: §8.3 exempts proof-only changes, and the diff touches nothing else.
  • npx tsc --noEmit exit 0.

@sergey-shandar
sergey-shandar marked this pull request as ready for review August 14, 2026 14:27
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@sergey-shandar
sergey-shandar added this pull request to the merge queue Aug 14, 2026
Merged via the queue into main with commit c3f30eb Aug 14, 2026
19 checks passed
@sergey-shandar
sergey-shandar deleted the claude/zen-archimedes-5s54pp branch August 14, 2026 20:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants