Skip to content

media/type: declare the magic-byte signatures once - #1533

Merged
sergey-shandar merged 5 commits into
mainfrom
claude/epic-fermi-i51u1h
Aug 13, 2026
Merged

media/type: declare the magic-byte signatures once#1533
sergey-shandar merged 5 commits into
mainfrom
claude/epic-fermi-i51u1h

Conversation

@sergey-shandar

Copy link
Copy Markdown
Contributor

Fixes fjs/media/type/todo/single-signature-table.md (deleted here).

Problem

Every recognized signature (PNG, JPEG, 2×GIF, PDF, 3×ZIP, WebP) was declared twice in fjs/media/type/module.f.mjs, in two representations that had to stay in byte-for-byte lockstep:

  • sentinel-Vec form for the pure path — table, plus the WebP special case riff / webp / isWebp, consumed by detect;
  • byte-pattern form for the streaming path — signatures (WebP's gap expressed as null wildcards), consumed by magicStep / detectVec / detectStream.

The comment above signatures admitted it: "The streaming counterpart of table/isWebp: the same signatures expressed as byte patterns…". Adding or correcting a signature meant editing both lists, and WebP was special-cased in both — a bespoke isWebp in one, a wildcard run in the other.

Change

signatures becomes the single source of truth. It is strictly the more general of the two forms: it already expresses WebP's four-byte size gap as wildcards, so no new helper is needed for the one non-contiguous signature.

detect is re-expressed on top of the eliminator that already existed:

export const detect = bytes => {
    /** @type {_MagicState} */
    let magic = magicInit
    for (const byte of iterable(u8List(msb)(bytes))) {
        magic = magicStep(magic, byte)
        if (magic.tag !== 'scan') { break }
    }
    return magicMime(magic)
}

table, sig, riff, webp and isWebp are deleted, along with the fromSentinel / startsWith / removeFront imports that only they used. The magic machinery (signatures, magicInit, magicStep, magicMime) moves above detect, since it is now shared rather than streaming-only.

The alternative the issue offered — deriving table from signatures by folding a wildcard-free pattern into a fromSentinel bigint to keep startsWith's prefix check — was not taken: it needs a new helper and keeps two runtime shapes of one list, for a detect whose only callers are its own proofs.

Behavior

detect's contract is unchanged, and fjs/media/type/proof.f.mjs passes unmodified — including its null cases (gif8NotGif, riffNotWebp, shortIsNull, emptyIsNull, textIsNull).

  • a matched signature is the answer (matched is absorbing, so trailing bytes are ignored exactly as startsWith ignored them);
  • an exhausted viable set is deadnull;
  • a prefix too short to complete any signature leaves the eliminator in scannull, which is the old "too short to match" semantics.

Cost is unchanged too: the fold breaks out as soon as the eliminator settles (≤12 bytes), so detect still does not walk a large Vec. Composing detect out of push/finish instead would have run the UTF-8 DFA over the whole blob on every non-matching input, so the magic factor is folded on its own.

One deliberate difference, now documented on detect: bytes are read through u8List, so a Vec whose bit length is not a whole number of bytes has its trailing partial byte zero-padded, where the old startsWith compared raw bits. That is the same reading of a ragged Vec push/detectVec already used, and CAS blobs are byte-aligned.

Docs

fjs/media/type/README.md — the signature table is labelled as documentation over the one signatures list, and the two places that described magicStep as "the streaming form of the table above" now name the shared list. fjs/media/type/todo/detect-json.md drops its link to the deleted issue and gets refreshed line references.

Checks

  • npx tsc — clean
  • npm start test — 2540 pass, 0 fail
  • npm run covmedia/type/module.f.mjs at 100.00 line / 100.00 branch / 100.00 function
  • npm run ci-update / npm run dev-update — no generated-file drift

No CHANGELOG entry yet — added in a follow-up commit on this branch once the PR number exists (AGENTS.md §8.3).

🤖 Generated with Claude Code

https://claude.ai/code/session_012CKmRpiXJ9rNGeQfotFVF9


Generated by Claude Code

`detect` eliminated a `Vec` against a sentinel-`Vec` `table` plus a bespoke
`isWebp`, while the streaming detector eliminated the same signatures against a
wildcard-capable byte-pattern list. Two representations of one mapping that had
to stay in byte-for-byte lockstep.

Keep `signatures`, the more general of the two — it already expresses WebP's
size gap as wildcards — and fold `detect`'s bytes through the same `magicStep`
eliminator, reading the verdict off with `magicMime`. `table`, `sig`, `riff`,
`webp` and `isWebp` are gone, along with the `fromSentinel` / `startsWith` /
`removeFront` imports they needed.

`detect`'s contract is unchanged: a matched signature wins, a dead eliminator
and a prefix too short to complete any signature both read as `null`.
`fjs/media/type/proof.f.mjs` passes unchanged, and the module keeps 100%
line/branch/function coverage.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CKmRpiXJ9rNGeQfotFVF9
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
functionalscript 6f9b1b5 Commit Preview URL

Branch Preview URL
Aug 13 2026, 09:09 PM

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CKmRpiXJ9rNGeQfotFVF9

@o2alexanderfedin o2alexanderfedin left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. Two things worth recording below — one is a real (deliberate, documented) behaviour change at a boundary, the other is a testing suggestion. Neither blocks.

Baseline: origin/main at 16142109; branch merge-base 373c3325, i.e. one commit behind, nothing overlapping.

Gates

  • npx tsc --noEmit — exit 0. npm run prepack from a clean tree — exit 0.
  • npm test2540 / 0, same as main.
  • Public surface: bin/extract.mjs and bin/consts.mjs identical to main; file-aware export type listing across the emitted .d.mts identical (589 lines both sides). signatures stays module-private, detect's signature is unchanged.
  • node bin/linkcheck.mjs — broken-link sets identical to main, so deleting todo/single-signature-table.md stranded nothing (the one inbound reference, in detect-json.md's Related list, is removed in the same commit). Its two updated line references check out: finish is at :221-229 and the UTF-8 factor at :140-161.
  • §8.3 CHANGELOG — code change, entry present, links only /pull/1533.

Faithfulness of the extraction, at the boundaries

I did not take "same signatures, one copy" on trust. I enumerated the nine signatures that existed before, compared them byte-for-byte against the new signatures list (they match, including the three PK variants and both GIF version headers, and WebP's four-byte gap becomes exactly four null wildcards), then ran main's detect and this head's detect side by side over 1115 constructed inputs: each signature exact, exact-plus-trailing-bytes, truncated by every whole number of bytes, truncated by 1–7 bits, one byte off at every position (±1 and ^0x80), shifted by a leading byte, the GIF8-prefix trap, WebP with RIFF only / size only / a wrong second marker / 11 bytes, plus 800 pseudo-random and signature-biased vectors at random bit lengths.

inputs=1115 divergent=1 nonNullMain=30 nonNullPr=31
  png/trunc-1bit  bytes=[89 50 4e 47 0d 0a 1a 0a] drop=1bit  main=null  pr=image/png
divergent with dropBits=0: 0

Negative control: flipping one byte of the JPEG pattern (0xff → 0xfe) on the PR side made the comparison report the expected extra divergences and then 0 again after reverting, so it is not a comparison that cannot fail.

So: on any byte-aligned Vec — the only shape a CAS blob or a file ever has — the new detect is byte-for-byte the old one. The single divergence is exactly the case the new JSDoc calls out: a Vec whose length is not a whole number of bytes now has its trailing partial byte zero-padded by u8List, so 71 bits of the PNG header complete the signature (PNG's last byte 0x0a has a zero low bit) where startsWith on a 72-bit pattern used to reject it.

That is the right direction, and it is a fix rather than a regression — I checked what main did with the same input:

main: detectVec(71bit).mime_type = image/png | detect(71bit) = null
pr:   detectVec(71bit).mime_type = image/png | detect(71bit) = image/png

main's two detectors disagreed on ragged input; this head makes them agree, which is the whole point of the extraction. Worth knowing that the old single-signature-table.md justified the change with "detectVec already proves the streaming machine reproduces detect's verdict on whole buffers" — true for whole buffers, and this is the one place where it was not. The CHANGELOG entry might be worth a clause saying sub-byte-length Vecs now read the same way detectStream reads them, since that is the only observable difference.

Proof coverage — a suggestion, not a regression

The proof (34 assertions) passes, but three mutations survive it:

  • deleting the PK 07 08 (spanned-archive) signature entirely;
  • magicStep's terminal-position test s.pattern.length === pos + 1pos + 2;
  • dropping the if (magic.tag !== 'scan') { break } in the new detect — this one is fine, since matched and dead are absorbing, so the mutant is semantically equivalent and surviving is correct.

The second is the one I would pin. The mutant makes every signature match one byte early, so it defeats precisely the guarantee the code comment claims: detect([47 49 46 38 37]) returns image/gif (the "GIF8"-prefix trap the comment says must not fire) and detect([ff d8]) returns image/jpeg.

I checked whether this is a regression before saying anything, and it is not: on main, deleting PK 07 08 from either copy, and the same pos + 2 mutation, all leave the proof green too. The gaps are pre-existing in the eliminator, and my 1115-input differential says the arithmetic is in fact correct today. What changes here is only that one untested line is now load-bearing for both detectors instead of one — which, on balance, is the extraction working as intended (a missing zip3 test used to have to be right in two places, now one). A truncated-signature case per pattern and a PK 07 08 case would close it cheaply. fromSentinel keeps its own coverage in bit_vec's proof, so dropping its last consumer here does not orphan it.

The README and module header now describe one list rather than two, the "table above is documentation" note is accurate, and the deleted todo's proposal matches what landed (table, the sig literals, riff, webp, isWebp all gone).

claude and others added 2 commits August 13, 2026 21:05
Review follow-up on #1533. Two mutants survived the proof: dropping the
spanned-ZIP (`PK 07 08`) signature, and shifting `magicStep`'s terminal test
from `pos + 1` to `pos + 2` so every signature matches one byte early — the
latter defeating the very "GIF8"-prefix guarantee the code comment claims.
Both gaps predate this PR, but the extraction makes the line load-bearing for
both detectors, so close them here: a `spannedZip` case and a `truncated` group
holding every signature one byte short. Verified both mutants now fail the
proof (7 and 1 failures) and the clean tree still passes.

Also note in the CHANGELOG the one observable difference: a `Vec` of sub-byte
length now reads the way `detectStream` already read it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CKmRpiXJ9rNGeQfotFVF9

Copy link
Copy Markdown
Contributor Author

Both items addressed in f250e2d (on top of your main merge, 720b519).

The surviving mutants. Closed the two that mattered:

  • spannedZipPK 07 08, so every entry of the signature list now has a case that fails if it is dropped.
  • a truncated group holding every signature one byte short (PNG at 7, JPEG at 2, both GIF headers at 5, PDF at 4, ZIP at 3, WebP at 11 — the last one also checks the wildcard run doesn't shorten the pattern).

Verified the mutants actually die rather than assuming it: pos + 1pos + 2 now fails 7 of 42, deleting PK 07 08 fails 1, and the clean tree passes 42/42. I left the break alone, agreeing with your reasoning — matched and dead are absorbing, so that mutant is semantically equivalent and surviving is the correct outcome.

Your framing of why these belong here rather than in a separate PR is the one I'd have used: the gaps are pre-existing in the eliminator, and the extraction is what makes the line load-bearing for both detectors.

CHANGELOG. Added the clause — the entry now ends "…so a Vec of sub-byte length reads the way detectStream already read it."

On the 71-bit PNG divergence: thank you for measuring what main did rather than just flagging the difference. I had reasoned about the zero-padding when writing the JSDoc note but had not checked that main's own detect and detectVec disagreed on that input — so I filed it as an acceptable cost where it is actually the two detectors converging. Worth having on the record.

Re-verified on the merged tree: npx tsc clean, npm start test 2548/0, media/type/module.f.mjs still 100.00 line / branch / function.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants