Skip to content

js/keywords: one source of truth for JavaScript keywords - #1562

Merged
sergey-shandar merged 5 commits into
mainfrom
claude/js-keywords
Aug 14, 2026
Merged

js/keywords: one source of truth for JavaScript keywords#1562
sergey-shandar merged 5 commits into
mainfrom
claude/js-keywords

Conversation

@sergey-shandar

Copy link
Copy Markdown
Contributor

Summary

FunctionalScript is a strict subset of JavaScript — any FunctionalScript program must run the same on JavaScript — so every consumer that decides whether a name is a keyword has to agree with the JavaScript grammar. Three hand-maintained copies existed: the JavaScript tokenizer's entry list, the DJS tokenizer's Set, and the rtti TypeScript printer's reserved-name screen (whose gaps took three review rounds on #1547 to close precisely because the list was hand-kept).

This carries the follow-up work that landed on the #1547 branch after that PR merged.

Key changes

  • New fjs/js/keywords module — the one source of truth, structured by the ECMA-262 categories:

    • reservedWords — the ReservedWord production (§12.7.2), never usable as identifiers;
    • strictModeReservedWordsimplements, interface, let, package, private, protected, public, static; every module is strict-mode code, so FunctionalScript treats them like reserved words;
    • restrictedNamesarguments/eval, not reserved words but unbindable in strict code;
    • keywords — the alphabetical aggregate, adding FunctionalScript's undefined (an ordinary global in JavaScript kept as a literal keyword).

    Two ties prevent drift: a type-level Assert<Equal<…>> pins the aggregate's literal union to the union of the groups, and the proof verifies it is exactly their sorted, duplicate-free union.

  • fjs/js/tokenizer derives keywordEntries from keywords, and _KeywordToken's kind is now Exclude<typeof keywords[number], 'true' | 'false' | 'null' | 'undefined'> — definitionally tied to the list instead of a parallel hand-written union, so a keyword added to the source of truth flows into the token type automatically.

  • fjs/djs/tokenizer builds its keyword Set from keywords instead of a copied literal.

  • fjs/types/rtti/ts composes its type-alias screen from reservedWords and strictModeReservedWords plus the TypeScript-only names (predefined type names, intrinsic, type-operator keywords), which correctly stay local — they are TypeScript facts, not JavaScript facts. The effective 61-name screen verified in RTTI TS printer: recursive schemas via the data form #1547's final review is unchanged.

Checks

  • npx tsc — clean.
  • Full test suite passes; fjs/js/keywords at 100% line/branch/function coverage; the tokenizers' coverage is unchanged.
  • No behavior change in any consumer: all three derived sets are equal to the previous hand-written ones.

🤖 Generated with Claude Code

https://claude.ai/code/session_01T8BR3aPUJDe5zZMDJKme7j


Generated by Claude Code

FunctionalScript is a strict subset of JavaScript — any FunctionalScript
program must run the same on JavaScript — so every consumer that decides
whether a name is a keyword has to agree with the JavaScript grammar.
Three hand-maintained copies existed: the JavaScript tokenizer's entry
list, the DJS tokenizer's Set, and the rtti TS printer's reserved list.

The new fjs/js/keywords module holds the sets once, structured by the
ECMA-262 categories: reservedWords (the ReservedWord production),
strictModeReservedWords (every module is strict-mode code),
restrictedNames (arguments/eval), and the aggregate keywords list that
adds FunctionalScript's undefined. A type-level pin ties the aggregate's
union to the groups' and the proof checks it is their sorted union.

- fjs/js/tokenizer derives keywordEntries from `keywords`, and
  _KeywordToken's kind is now Exclude<typeof keywords[number], ...> —
  definitionally tied to the list instead of a parallel hand-written
  union.
- fjs/djs/tokenizer builds its Set from `keywords`.
- fjs/types/rtti/ts composes its type-alias screen from reservedWords
  and strictModeReservedWords plus the TypeScript-only names; the
  effective set is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T8BR3aPUJDe5zZMDJKme7j
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T8BR3aPUJDe5zZMDJKme7j
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
functionalscript fd09f75 Commit Preview URL

Branch Preview URL
Aug 14 2026, 09:51 PM

Comment thread fjs/djs/tokenizer/module.f.mjs Outdated
'public', 'return', 'static', 'super', 'switch', 'this', 'throw', 'try',
'typeof', 'var', 'void', 'while', 'with', 'yield',
]))
const keywordSet = /** @type {ReadonlySet<string>} */ (new Set(keywords))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Avoid using /** @type {T} */(value). Use either /** @type {T} */const ... = ... or @satisfies. See AGENTS.md for that.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 65649da — declaration form, so the compiler checks new Set(keywords) against the annotation instead of overriding it.

For the record, one inline @type cast remains in this PR, on js/tokenizer's derived keywordEntries (/** @type {JsToken} */ ({ kind })). The declaration form (and @satisfies) is a compile error there: { kind: <49-literal union> } does not distribute over the JsToken discriminated union, which is exactly why the cast exists. It is the same boundary cast the file's djs counterpart toJsToken already uses for the same reason, and it is safe by construction — _KeywordToken's kinds derive from the same keywords list.


Generated by Claude Code

Per AGENTS.md, the declaration form lets the compiler check the
initializer against the annotation instead of overriding it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T8BR3aPUJDe5zZMDJKme7j

@o2alexanderfedin o2alexanderfedin left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. Reviewed at 8441bd5, baseline origin/main = c3af487f ("Add task to split AGENTS.md into scoped documents (#1560)").

The risk in a DRY extraction like this is that the consumers were not actually identical — #1522's hex-codec extraction found DJS deliberately divergent, and #1533's sweep found a real divergence hiding as a latent bug. So I enumerated all three consumers' sets before and diffed them against the new shared lists.

Set diff — the three consumers, before vs after

Old sets parsed out of origin/main's sources; new sets taken from the exported lists (plus each consumer's own extras):

consumer old new removed added
js/tokenizer keywordEntries 49 49
djs/tokenizer keywords 49 49
types/rtti/ts reserved 61 61

No set difference anywhere. No duplicates on either side. keywords is exactly reservedWords ∪ strictModeReservedWords ∪ restrictedNames ∪ {undefined}, sorted (38 + 8 + 2 + 1 = 49).

The rtti regrouping is the one that needed care and it lands correctly: false/null/true/void move out of its "predefined type names" comment-group into reservedWords, and yield moves out of its strict-mode group (the new strictModeReservedWords has 8 entries, no yield) into reservedWords — net zero. undefined correctly stays in rtti's own list, since reservedWords does not contain it and rtti does not spread restrictedNames. Re #1547: all nine TS strict-mode reserved words and intrinsic are present and accounted for.

Behavioural sweep across all three consumers

Set equality is not the same as behaviour equality, so I also drove each consumer's real entry point over 100 words — every keyword old and new, the TS predefined type names and type operators, TS contextual keywords that were never in any list (as, is, of, get, set, async, from, satisfies, asserts, declare, namespace, type, abstract, accessor, override, using, …), and near-miss identifiers (Await, AWAIT, awaits, yield2, let2, _, $, T0, …):

  • js/tokenizer tokenize — full token stream, JSON-compared
  • djs/tokenizer tokenizeJs — full token stream, JSON-compared
  • types/rtti/ts dataToTs — whether a rule of that name keeps its identifier or gets a generated T<n>

Identical on origin/main and at this head for all 300 results. Negative control (perturbing yield's three results) diffs as expected, so the harness discriminates.

Type level

_KeywordToken went from a hand-written six-arm union to Exclude<typeof keywords[number], 'true' | 'false' | 'null' | 'undefined'>. I checked that against main's literal union with a [A] extends [B] ? [B] extends [A] equality inside the project's own tsc: exactly equal; adding one phantom kind to the old union makes it fail (TS2322), so the check is real.

_KeywordsPinned is not decorative either — appending 'zzz' to keywords gives module.f.mjs(65,21): error TS2344: Type 'false' does not satisfy the constraint 'true'. Good: it means the /** @type {JsToken} */ cast in the new keywords.map(…) cannot silently drift, because _KeywordToken is derived from the same list the cast reads.

Gates

  • npx tsc --noEmit → 0. npm run prepack from a clean tree → 0.
  • npm testpass: 2709, fail: 0 vs main's 2708 — exactly the one new js/keywords proof.
  • Emitted declarations read directly (not bin/extract.mjs): js/tokenizer/module.f.d.mts, djs/tokenizer/module.f.d.mts and types/rtti/ts/module.f.d.mts are byte-identical to main. The only surface change is the new js/keywords module itself, whose consts emit as precise readonly [...] tuples (not string[]). @module survives in the new declaration (1 occurrence, blank line after the header present); the single : any grep hit is the word "any" in prose.
  • bin/linkcheck.mjs — broken-link sets identical to main.

Proof quality

proof.aggregate mutation-tested rather than trusted by name:

mutation result
drop 'yield' from reservedWords fail: 1
swap 'eval'/'export' in keywords (order only) fail: 1
drop 'class' from keywords fail: 3 — the aggregate proof plus two tokenizer proofs

The third one is the useful one: it shows the DRY wiring is real, not just declared — a change to the shared list propagates into both tokenizers' behaviour. And pinning the sort order matters, since keywordEntries feeds ordered_map's fromEntries.

Minor

changelog/unreleased/1562.md ends with [#1562](…/pull/1562), but §8.3 now says entries carry "no PR number or link inside the file". Same drift I noted on #1564; most existing unreleased/ files still have links, so this follows the de-facto shape rather than the written rule.

Not an issue

  • types.ts importing import type { keywords } from a .f.mjs introduces no cycle — js/keywords imports nothing from js/tokenizer.
  • reserved in rtti widening from a const tuple to readonly string[] is invisible: it is module-local and the emitted declaration is unchanged.
  • _KeywordsPinned is _-prefixed, correct under §6.2 (a JSDoc @typedef has no non-exported form).
  • No README.md for the new module directory — fjs/js/tokenizer has none either, and I find no rule requiring one.

The consumers really were identical this time, and the extraction is faithful to all three.

@sergey-shandar
sergey-shandar added this pull request to the merge queue Aug 14, 2026
Merged via the queue into main with commit 4a3b687 Aug 14, 2026
19 checks passed
@sergey-shandar
sergey-shandar deleted the claude/js-keywords branch August 14, 2026 22:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants