media/type: declare the magic-byte signatures once - #1533
Conversation
`detect` eliminated a `Vec` against a sentinel-`Vec` `table` plus a bespoke `isWebp`, while the streaming detector eliminated the same signatures against a wildcard-capable byte-pattern list. Two representations of one mapping that had to stay in byte-for-byte lockstep. Keep `signatures`, the more general of the two — it already expresses WebP's size gap as wildcards — and fold `detect`'s bytes through the same `magicStep` eliminator, reading the verdict off with `magicMime`. `table`, `sig`, `riff`, `webp` and `isWebp` are gone, along with the `fromSentinel` / `startsWith` / `removeFront` imports they needed. `detect`'s contract is unchanged: a matched signature wins, a dead eliminator and a prefix too short to complete any signature both read as `null`. `fjs/media/type/proof.f.mjs` passes unchanged, and the module keeps 100% line/branch/function coverage. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012CKmRpiXJ9rNGeQfotFVF9
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
functionalscript | 6f9b1b5 | Commit Preview URL Branch Preview URL |
Aug 13 2026, 09:09 PM |
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012CKmRpiXJ9rNGeQfotFVF9
o2alexanderfedin
left a comment
There was a problem hiding this comment.
Approving. Two things worth recording below — one is a real (deliberate, documented) behaviour change at a boundary, the other is a testing suggestion. Neither blocks.
Baseline: origin/main at 16142109; branch merge-base 373c3325, i.e. one commit behind, nothing overlapping.
Gates
npx tsc --noEmit— exit 0.npm run prepackfrom a clean tree — exit 0.npm test— 2540 / 0, same asmain.- Public surface:
bin/extract.mjsandbin/consts.mjsidentical tomain; file-awareexport typelisting across the emitted.d.mtsidentical (589 lines both sides).signaturesstays module-private,detect's signature is unchanged. node bin/linkcheck.mjs— broken-link sets identical tomain, so deletingtodo/single-signature-table.mdstranded nothing (the one inbound reference, indetect-json.md's Related list, is removed in the same commit). Its two updated line references check out:finishis at:221-229and the UTF-8 factor at:140-161.- §8.3 CHANGELOG — code change, entry present, links only
/pull/1533.
Faithfulness of the extraction, at the boundaries
I did not take "same signatures, one copy" on trust. I enumerated the nine signatures that existed before, compared them byte-for-byte against the new signatures list (they match, including the three PK variants and both GIF version headers, and WebP's four-byte gap becomes exactly four null wildcards), then ran main's detect and this head's detect side by side over 1115 constructed inputs: each signature exact, exact-plus-trailing-bytes, truncated by every whole number of bytes, truncated by 1–7 bits, one byte off at every position (±1 and ^0x80), shifted by a leading byte, the GIF8-prefix trap, WebP with RIFF only / size only / a wrong second marker / 11 bytes, plus 800 pseudo-random and signature-biased vectors at random bit lengths.
inputs=1115 divergent=1 nonNullMain=30 nonNullPr=31
png/trunc-1bit bytes=[89 50 4e 47 0d 0a 1a 0a] drop=1bit main=null pr=image/png
divergent with dropBits=0: 0
Negative control: flipping one byte of the JPEG pattern (0xff → 0xfe) on the PR side made the comparison report the expected extra divergences and then 0 again after reverting, so it is not a comparison that cannot fail.
So: on any byte-aligned Vec — the only shape a CAS blob or a file ever has — the new detect is byte-for-byte the old one. The single divergence is exactly the case the new JSDoc calls out: a Vec whose length is not a whole number of bytes now has its trailing partial byte zero-padded by u8List, so 71 bits of the PNG header complete the signature (PNG's last byte 0x0a has a zero low bit) where startsWith on a 72-bit pattern used to reject it.
That is the right direction, and it is a fix rather than a regression — I checked what main did with the same input:
main: detectVec(71bit).mime_type = image/png | detect(71bit) = null
pr: detectVec(71bit).mime_type = image/png | detect(71bit) = image/png
main's two detectors disagreed on ragged input; this head makes them agree, which is the whole point of the extraction. Worth knowing that the old single-signature-table.md justified the change with "detectVec already proves the streaming machine reproduces detect's verdict on whole buffers" — true for whole buffers, and this is the one place where it was not. The CHANGELOG entry might be worth a clause saying sub-byte-length Vecs now read the same way detectStream reads them, since that is the only observable difference.
Proof coverage — a suggestion, not a regression
The proof (34 assertions) passes, but three mutations survive it:
- deleting the
PK 07 08(spanned-archive) signature entirely; magicStep's terminal-position tests.pattern.length === pos + 1→pos + 2;- dropping the
if (magic.tag !== 'scan') { break }in the newdetect— this one is fine, sincematchedanddeadare absorbing, so the mutant is semantically equivalent and surviving is correct.
The second is the one I would pin. The mutant makes every signature match one byte early, so it defeats precisely the guarantee the code comment claims: detect([47 49 46 38 37]) returns image/gif (the "GIF8"-prefix trap the comment says must not fire) and detect([ff d8]) returns image/jpeg.
I checked whether this is a regression before saying anything, and it is not: on main, deleting PK 07 08 from either copy, and the same pos + 2 mutation, all leave the proof green too. The gaps are pre-existing in the eliminator, and my 1115-input differential says the arithmetic is in fact correct today. What changes here is only that one untested line is now load-bearing for both detectors instead of one — which, on balance, is the extraction working as intended (a missing zip3 test used to have to be right in two places, now one). A truncated-signature case per pattern and a PK 07 08 case would close it cheaply. fromSentinel keeps its own coverage in bit_vec's proof, so dropping its last consumer here does not orphan it.
The README and module header now describe one list rather than two, the "table above is documentation" note is accurate, and the deleted todo's proposal matches what landed (table, the sig literals, riff, webp, isWebp all gone).
Review follow-up on #1533. Two mutants survived the proof: dropping the spanned-ZIP (`PK 07 08`) signature, and shifting `magicStep`'s terminal test from `pos + 1` to `pos + 2` so every signature matches one byte early — the latter defeating the very "GIF8"-prefix guarantee the code comment claims. Both gaps predate this PR, but the extraction makes the line load-bearing for both detectors, so close them here: a `spannedZip` case and a `truncated` group holding every signature one byte short. Verified both mutants now fail the proof (7 and 1 failures) and the clean tree still passes. Also note in the CHANGELOG the one observable difference: a `Vec` of sub-byte length now reads the way `detectStream` already read it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012CKmRpiXJ9rNGeQfotFVF9
|
Both items addressed in The surviving mutants. Closed the two that mattered:
Verified the mutants actually die rather than assuming it: Your framing of why these belong here rather than in a separate PR is the one I'd have used: the gaps are pre-existing in the eliminator, and the extraction is what makes the line load-bearing for both detectors. CHANGELOG. Added the clause — the entry now ends "…so a On the 71-bit PNG divergence: thank you for measuring what Re-verified on the merged tree: Generated by Claude Code |
Fixes
fjs/media/type/todo/single-signature-table.md(deleted here).Problem
Every recognized signature (PNG, JPEG, 2×GIF, PDF, 3×ZIP, WebP) was declared twice in
fjs/media/type/module.f.mjs, in two representations that had to stay in byte-for-byte lockstep:Vecform for the pure path —table, plus the WebP special caseriff/webp/isWebp, consumed bydetect;signatures(WebP's gap expressed asnullwildcards), consumed bymagicStep/detectVec/detectStream.The comment above
signaturesadmitted it: "The streaming counterpart oftable/isWebp: the same signatures expressed as byte patterns…". Adding or correcting a signature meant editing both lists, and WebP was special-cased in both — a bespokeisWebpin one, a wildcard run in the other.Change
signaturesbecomes the single source of truth. It is strictly the more general of the two forms: it already expresses WebP's four-byte size gap as wildcards, so no new helper is needed for the one non-contiguous signature.detectis re-expressed on top of the eliminator that already existed:table,sig,riff,webpandisWebpare deleted, along with thefromSentinel/startsWith/removeFrontimports that only they used. The magic machinery (signatures,magicInit,magicStep,magicMime) moves abovedetect, since it is now shared rather than streaming-only.The alternative the issue offered — deriving
tablefromsignaturesby folding a wildcard-free pattern into afromSentinelbigint to keepstartsWith's prefix check — was not taken: it needs a new helper and keeps two runtime shapes of one list, for adetectwhose only callers are its own proofs.Behavior
detect's contract is unchanged, andfjs/media/type/proof.f.mjspasses unmodified — including itsnullcases (gif8NotGif,riffNotWebp,shortIsNull,emptyIsNull,textIsNull).matchedis absorbing, so trailing bytes are ignored exactly asstartsWithignored them);dead→null;scan→null, which is the old "too short to match" semantics.Cost is unchanged too: the fold breaks out as soon as the eliminator settles (≤12 bytes), so
detectstill does not walk a largeVec. Composingdetectout ofpush/finishinstead would have run the UTF-8 DFA over the whole blob on every non-matching input, so the magic factor is folded on its own.One deliberate difference, now documented on
detect: bytes are read throughu8List, so aVecwhose bit length is not a whole number of bytes has its trailing partial byte zero-padded, where the oldstartsWithcompared raw bits. That is the same reading of a raggedVecpush/detectVecalready used, and CAS blobs are byte-aligned.Docs
fjs/media/type/README.md— the signature table is labelled as documentation over the onesignatureslist, and the two places that describedmagicStepas "the streaming form of the table above" now name the shared list.fjs/media/type/todo/detect-json.mddrops its link to the deleted issue and gets refreshed line references.Checks
npx tsc— cleannpm start test— 2540 pass, 0 failnpm run cov—media/type/module.f.mjsat 100.00 line / 100.00 branch / 100.00 functionnpm run ci-update/npm run dev-update— no generated-file driftNo CHANGELOG entry yet — added in a follow-up commit on this branch once the PR number exists (AGENTS.md §8.3).
🤖 Generated with Claude Code
https://claude.ai/code/session_012CKmRpiXJ9rNGeQfotFVF9
Generated by Claude Code