Give `fjs/text/ascii` the hex-digit codec its three consumers were duplicating by sergey-shandar · Pull Request #1522 · functionalscript/functionalscript

sergey-shandar · 2026-08-13T10:35:41Z

Implements fjs/text/ascii/todo/hex-digit-codec.md (P3, open, no blockers), deleted here.

Problem

One mapping — hex-digit code point ↔ value, i.e. the offsets digit0, latinSmallLetterA - 10, latinCapitalLetterA - 10 — was written three times, in both directions:

site	direction	shape
`fjs/media/json/serializer`	value → char	`value < 10 ? digit0 + value : latinSmallLetterA + value - 10`
`fjs/js/tokenizer`	char → value	range-map dispatch, one `parseUnicodeCharHex(offset)` per range
`fjs/djs/tokenizer`	char → value	ternary chain over its own `AF` range

The af / AF ranges were built independently in the two tokenizers as well. The djs copy's fallthrough assumed lowercase without checking, so a non-hex code point decoded to a garbage digit where the js copy had a real reject path.

Change

fjs/text/ascii already owns digit0, latinSmallLetterA, latinCapitalLetterA (and latinSmallLetterF / latinCapitalLetterF, which exist for no other reason), so it now owns the codec:

hexDigitValue: (codePoint: number) => Nullable<number> — value 0..15, null for anything that is not 0-9 / a-f / A-F;
hexDigitCodePoint: (value: number) => number — the lowercase digit for a value in 0..15;
latinSmallLetterAFRange / latinCapitalLetterAFRange beside the existing digitRange.

Both offsets are computed once, and hexDigitCodePoint is written in terms of the same latinSmallLetterAFOffset that hexDigitValue inverts.

All three consumers now call it:

serializer — hexDigit is fromCharCode(hexDigitCodePoint(value)).
js/tokenizer — the unicodeChar state drops its three-range dispatch entirely. hexDigitValue classifies and decodes in one step, and null is exactly the non-hex input parseUnicodeCharDefault already rejected with 'invalid hex value', so both the accept and reject paths stay covered by the existing proofs.
djs/tokenizer — mapUnwrap(hexDigitValue), following the try* + mapUnwrap precedent in fjs/text and fjs/types/bit_vec. A \uXXXX escape reaches that state only after the grammar accepted its four hex digits, so the impossible case now asserts rather than decoding to garbage — and the assert branch lives in the shared helper, adding no uncovered branch at the call site.

No public export is removed; everything deleted was module-private, so this is additive.

Checks

npx tsc — clean.
npm start test — 2530 pass, 0 fail.
npm run cov — fjs/text/ascii/module.f.mjs at 100% line/branch/function; no new uncovered line, branch, or function anywhere. The two tokenizers' percentages move by hundredths purely because covered branches/functions were removed (djs 96.89 → 96.83 branch, js 97.53 → 97.54 branch / 97.42 → 97.40 function).
npm run ci-update — no diff.

Rust untouched.

Generated by Claude Code

The hex-digit ↔ value mapping — the `'0'`, `'a' - 10` and `'A' - 10` offsets — was written three times: encode-side in the JSON serializer, decode-side as a range-map dispatch in `js/tokenizer`, and decode-side as a ternary chain in `djs/tokenizer`, which built its own `AF` range and assumed lowercase in the fallthrough (a non-hex code point decoded to a garbage digit). `fjs/text/ascii` already owns the constants, so it now owns the codec too: `hexDigitValue` (`Nullable`, `null` for a non-hex code point), `hexDigitCodePoint` (lowercase), plus the `a-f` / `A-F` ranges. All three consumers use it. `js/tokenizer`'s `unicodeChar` state no longer needs its three-range dispatch — a `null` value is exactly the input the default handler rejects — and `djs/tokenizer` wraps the codec with `mapUnwrap`, so the grammar-guaranteed invariant asserts instead of decoding garbage. Closes the `fjs/text/ascii/todo/hex-digit-codec.md` issue. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CWo5y1CqQ8fgVSn3XHEM8W

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CWo5y1CqQ8fgVSn3XHEM8W

cloudflare-workers-and-pages · 2026-08-13T10:35:58Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	functionalscript	`1fdbed3`	Commit Preview URL Branch Preview URL	Aug 13 2026, 11:02 AM

o2alexanderfedin

Approving. This is a clean extraction, and the DJS fix that came with it is the real prize.

Gates

npx tsc --noEmit → 0
npm test → 2530 pass / 0 fail vs 2524 on origin/main; the delta is exactly the 6 new proof cases, no test lost
npm run prepack from a clean tree → 0
Public surface diff (extract.mjs 46 KB, consts.mjs 65 KB both sides, so the tools produced real output): zero type-alias changes; consts gain exactly four entries, all in fjs/text/ascii — latinCapitalLetterAFRange: Range, latinSmallLetterAFRange: Range, hexDigitValue: (codePoint: number) => Nullable<number>, hexDigitCodePoint: (value: number) => number. No unprefixed export type added, nothing widened to any.
Broken-link sets identical to main (140 both, empty set diff); no dangling reference to the deleted todo/hex-digit-codec.md anywhere in the tree.
@module survives declaration emit in fjs/text/ascii/module.f.d.mts (§4 blank line intact) even with the new @import tags after it.
The new import { contains } from '../../types/range/module.f.mjs' introduces no cycle — fjs/types/range/module.f.mjs has no imports at all — and every const consumed at module-eval time (digitRange, latinSmallLetterA, latinCapitalLetterA) is defined above its use.

Behavioural equivalence at the three call sites

I compared old against new rather than trusting the extraction, exhaustively over code points 0..0x2000:

fjs/js/tokenizer — the old create(parseUnicodeCharDefault)([rangeFunc(digitRange)…, rangeFunc(rangeSmallAF)…, rangeFunc(rangeCapitalAF)…]) dispatch vs the new hexDigitValue(input) === null ? default : …: 0 mismatches, treating null as "fell through to the default handler". The 'invalid hex value' path is still exercised by fjs/js/tokenizer/proof.f.mjs:163.
fjs/media/json/serializer — value < 10 ? digit0 + value : latinSmallLetterA + value - 10 vs hexDigitCodePoint: 0 mismatches over 0..15, and they also agree outside the documented domain (16 → 0x67 in both), so nothing changes even for a caller that was out of contract.
fjs/djs/tokenizer — not equivalent, deliberately, and this is the good part. The old chain's final else was unguarded: any non-hex code point reaching the unicode state decoded to cp - 87, so cp = 0 produced -87 and the accumulator absorbed it silently. mapUnwrap(hexDigitValue) now asserts instead. I checked the unreachability claim in the header comment rather than taking it: the token grammar at :74-92 spells the escape as 'u', ...repeat(4)({ digit, AF: range('AF'), af: range('af') }), so the four hex digits are grammar-accepted before decodeJsonString runs. The assert is genuinely defensive, and the comment says exactly that.

Boundary spot-check on the new function: '0'→0, '9'→9, 'a'→10, 'f'→15, 'A'→10, 'F'→15, and '/' ':' '@' 'G' '`' 'g' → null.

Proof (§3.2)

Mutation-tested the new proof, eight mutations, all killed:

mutation	killed by
`codePoint - digit0` → `+ 1`	`hexDigitValue.digit`, `hexDigitCodePoint.roundTrip`
lowercase offset off-by-one	`hexDigitValue.latinSmallLetterAF`, `roundTrip`
uppercase offset off-by-one	`hexDigitValue.latinCapitalLetterAF`
`: null` → `: 0`	`hexDigitValue.notAHexDigit`
`value < 10` → `value <= 10`	`hexDigitCodePoint.lowercaseDigits`, `roundTrip`
`range('af')` → `range('ae')`	`hexDigitValue.latinSmallLetterAF`, `roundTrip`
`range('AF')` → `range('AE')`	`hexDigitValue.latinCapitalLetterAF`
`isDigit(codePoint) ?` → `false ?`	`hexDigitValue.digit`, `roundTrip`

One nit, not a blocker

notAHexDigit pins five of the six boundary neighbours — '/' (0x2F), ':' (0x3A), '@' (0x40), 'G' (0x47), 'g' (0x67) — but not '`' (0x60), the code point just below 'a'. That is the one mutation that survives: widening latinSmallLetterAFRange to [0x60, 0x66] leaves all 8 cases green. Behaviour is correct today; it is only the lower bound of the lowercase range that is unpinned, and one more assertEq(hexDigitValue(one('\')), null)` closes it.

CHANGELOG entry is well-formed — code change, links only /pull/1522, and it names the DJS behaviour change explicitly rather than burying it.

`notAHexDigit` covered five of the six boundary neighbours; the missing one (0x60, just below 'a') left `latinSmallLetterAFRange`'s lower bound unpinned, so widening it to [0x60, 0x66] kept every case green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CWo5y1CqQ8fgVSn3XHEM8W

o2alexanderfedin

Re-approving at 1fdbed37 — the head moved right after my previous review, adding exactly the missing boundary case.

Re-verified at the new head:

npx tsc --noEmit → 0; npm test → 2530 pass / 0 fail, unchanged
The mutation that survived before now dies: widening latinSmallLetterAFRange to [0x60, 0x66] fails hexDigitValue.notAHexDigit, where previously all 8 cases stayed green

Everything else in my review at 04d36a89 stands — the three call sites are unchanged by this commit, so the equivalence measurements and the surface diff carry over.

claude added 2 commits August 13, 2026 10:35

Add CHANGELOG entry for #1522

04d36a8

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CWo5y1CqQ8fgVSn3XHEM8W

o2alexanderfedin approved these changes Aug 13, 2026

View reviewed changes

sergey-shandar added this pull request to the merge queue Aug 13, 2026

Merged via the queue into main with commit 095adee Aug 13, 2026
19 checks passed

o2alexanderfedin mentioned this pull request Aug 13, 2026

Re-normalize round53's rounded mantissa back into 53 bits #1524

Closed

sergey-shandar deleted the claude/epic-fermi-ux8ixo branch August 13, 2026 21:10

This was referenced Aug 14, 2026

js/keywords: one source of truth for JavaScript keywords #1562

Merged

fsm: build toRange on ascii.range instead of re-implementing it #1575

Merged

Navigation Menu

Uh oh!

Uh oh!

Give `fjs/text/ascii` the hex-digit codec its three consumers were duplicating - #1522

Give `fjs/text/ascii` the hex-digit codec its three consumers were duplicating#1522
sergey-shandar merged 3 commits into
mainfrom
claude/epic-fermi-ux8ixo

sergey-shandar commented Aug 13, 2026

Uh oh!

cloudflare-workers-and-pages Bot commented Aug 13, 2026 •

edited

Loading

Uh oh!

o2alexanderfedin left a comment

Uh oh!

o2alexanderfedin left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

sergey-shandar commented Aug 13, 2026

Problem

Change

Checks

Uh oh!

cloudflare-workers-and-pages Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying with Cloudflare Workers

Uh oh!

o2alexanderfedin left a comment

Choose a reason for hiding this comment

Gates

Behavioural equivalence at the three call sites

Proof (§3.2)

One nit, not a blocker

Uh oh!

o2alexanderfedin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cloudflare-workers-and-pages Bot commented Aug 13, 2026 •

edited

Loading