From bbc64335ee6f516df69a80562b0a9ebc980ac8ed Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 28 Aug 2026 20:40:02 +0000 Subject: [PATCH 01/14] todo: plan the JSON/DataJS/FunctionalScript restructure A coordinating issue for restructuring the parser/serializer stack into three tiers: a self-contained JSON codec, a new spec'd DataJS interchange format (JSON extended from tree to DAG, nothing else) in fjs/media/datajs, and the current fjs/djs front end moving to fjs/fsc to grow with the language. Records the design decision log (';'-terminated consts, JSON whitespace, JS-derived duplicate-key semantics, special-number round-trips, ASCII const names, subset laws DataJS < FJS < JS), the staged migration sequence, and the edits owed to the five existing issues it supersedes or rebases. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho --- todo/parser-serializer-restructure.md | 261 ++++++++++++++++++++++++++ 1 file changed, 261 insertions(+) create mode 100644 todo/parser-serializer-restructure.md diff --git a/todo/parser-serializer-restructure.md b/todo/parser-serializer-restructure.md new file mode 100644 index 000000000..4f656c20f --- /dev/null +++ b/todo/parser-serializer-restructure.md @@ -0,0 +1,261 @@ +## Restructure JSON, DataJS, and FunctionalScript parsers/serializers + +**Priority:** P2 +**Status:** open + +This is a coordinating issue: it records the design decided in discussion, +sequences the stages, and names the edits owed to existing issues. Each stage +gets its own co-located `todo/` file when it starts; concrete tasks live there, +not here. + +### Problem + +Parsing and serialization are spread over four module families whose +relationships grew rather than being designed: + +- **`fjs/media/json`** — the structural parser (one container machine with a + `NumberPolicy` seam, standard and extended codecs) and serializer are JSON's + own and recently reworked. Its *tokenizer* is not: it is a ~100-line adapter + over `fjs/js/tokenizer`, the hand-written 747-line JavaScript tokenizer. +- **`fjs/djs`** — a full module pipeline: a grammar-based BNF tokenizer, a BNF + parser over token symbols, `AstModule`, and the transpiler behind + `fjs compile`. It conflates two different things: a data interchange format + (values, `const` sharing) and the language front end (imports, comments, + identifier keys, future expressions). +- **`fjs/fsc`** — nearly empty: a character-classifier stub plus a dead third + copy of the JSON grammar + ([orphaned-json-grammar](../fjs/fsc/todo/orphaned-json-grammar.md)). +- **`fjs/bnf`** — the grammar toolkit, still evolving (EOF encoding change, + pending unicode split). + +Two structural problems follow: + +1. **The media codecs sit downstream of permanently evolving code.** JSON and + the data format depend on the JS token vocabulary (`fjs/js/tokenizer`), + which must grow with FunctionalScript. A frozen interchange format cannot + be built on a mutating lexer, and the same argument bars a runtime + dependency on `fjs/bnf` until that module is stable. +2. **The data format and the compiler front end are one codebase.** The DJS + pipeline cannot be promoted to a spec'd, "implement it in an afternoon" + format while it also carries module framing, trivia, and the growth path of + the language. + +### Proposal + +Three tiers, each self-contained, with dependency arrows pointing only at +spec-frozen layers: + +```text +fjs/media/json own tokenizer + parser frozen by the JSON spec + ▲ +fjs/media/datajs reuses JSON's pieces frozen by the DataJS spec + (strings, numbers, containers) + +fjs/fsc JS tokenizer (comments, all evolves with the language + operators) → parser → AST → EDAG +``` + +- **JSON**: accepted language and value semantics are frozen; the tokenizer + becomes self-contained (the `fjs/js/tokenizer` wrapper is replaced by a + small scanner of JSON's own lexical grammar). Error shapes may change once + in that swap; accepted-input behavior and proofs do not. +- **DataJS** (the format known in this repository as DJS): a new, minimal, + spec'd format — JSON extended from a tree to a DAG, nothing else. New + hand-written parser and serializer in `fjs/media/datajs`, layered on JSON's + exported pieces. Everything that is not needed for the DAG property moves to + FunctionalScript. +- **FunctionalScript**: the current `fjs/djs` front end (grammar-based + tokenizer, BNF parser, AST, transpiler) moves to `fjs/fsc` and continues to + grow there — comments, imports, identifier keys, and the staged EDAG work. + The compiler can emit DataJS (normalized) or JSON. +- **BNF is not a runtime dependency of the media codecs.** The spec carries + the grammars as BNF text; `fjs/bnf/**` may hold the JSON and DataJS grammars + as *proof-covered examples* cross-checked against the spec's test vectors. + An example grammar without proof coverage is how + [orphaned-json-grammar](../fjs/fsc/todo/orphaned-json-grammar.md) happened; + none may be added without proofs. + +### The DataJS format (decision record) + +Decisions made in design discussion; the spec (stage 1) is their normative +home. The governing principle: **derive behavior from JS**. DataJS ⊂ +FunctionalScript ⊂ JavaScript, where `⊂` means *accepted with identical +meaning* — a subset may reject what its superset accepts, but must never +accept something and mean something different by it. + +**Name.** DataJS; "DJS" survives only as an informal abbreviation. Not +"DataScript" (taken by a well-known database library). The npm name `datajs` +belongs to a defunct OData library — check availability before any standalone +package publishes; the spec does not need it. + +**Data model.** A DAG of values. Leaves are JSON's primitives plus `bigint`, +`undefined`, `NaN`, `Infinity`, `-Infinity`, `-0`. Number round-trips satisfy +`Object.is`. Object entries follow JS duplicate-key semantics exactly: value +from the last occurrence, position from the first. Sharing is semantic — two +references to one `const` denote the same node, and references may only point +at *earlier* consts, so a document is acyclic by construction and parseable in +one pass. The reference parser returns live JS values and does not freeze them +(FunctionalScript has no `Object.freeze`); the spec is silent on freezing and +other implementations may. + +**Syntax.** + +```text +module ::= const* export +const ::= 'const' id '=' value ';' +export ::= 'export' 'default' value (no trailing ';') +value ::= primitive | id | array | object +key ::= string | '[' '"__proto__"' ']' +``` + +- **`;` terminates every `const`;** no `;` after `export default`, no empty + statements. Rationale, each sufficient alone: no line-terminator taxonomy in + the spec (a lone CR *is* a JS `LineTerminator` — trivia no implementer + should need); one canonical spelling per document; the separator is a + visible character, so byte-different files that render identically cannot + differ in meaning; and a document minifies to one line — + `const a=[];export default[a,a]` — enabling DataJS inside JSON strings, + line-delimited streaming, and one-line test fixtures. Whitespace is needed + only between adjacent word-tokens (`const a`, `export default x`). +- **Whitespace is JSON's** — space, tab, LF, CR — insignificant everywhere. + Other JS whitespace (U+2028/U+2029, NBSP, FF, BOM) is rejected. +- **No comments, no imports.** A DataJS document is closed; the compiler + inlines resolved imports when normalizing FunctionalScript to DataJS. +- **Strings and numbers are JSON's grammar**, plus the bigint `n` suffix. + `-` is not an operator: it folds into a following number, bigint, or + `Infinity` token only (`-NaN`, `-undefined`, a bare `-` are rejected). +- **Keys** are JSON strings, plus the computed spelling `["__proto__"]` as the + only way to write that one key; a bare or string `"__proto__"` key is + rejected (JS would read it as prototype replacement). +- **Const names** are ASCII: `[A-Za-z_$][A-Za-z0-9_$]*`, each bound once, + and binding `undefined`, `NaN`, or `Infinity` is rejected — JS permits + `const undefined = 5` and later `undefined` then means the const, which a + subset treating it as a literal would silently reinterpret. +- **A JSON document is a valid DataJS value, never a DataJS document** (a + DataJS document is a JS module, so it cannot be a JSON document). The + conversion is literal: `"export default " + json + ";"` — minus the `;`, + `"export default " + json` — is always a valid document. + +**Serialization.** Any conforming serializer may emit any valid document; a +separate *normalized form* section defines one byte-deterministic canonical +serializer (const names `_0`, `_1`, … in first-emission order; a const emitted +iff its value is referenced more than once; shortest round-trip number +spelling; bigints as full digits + `n`; fixed string escaping). Normalization +is not a blocker for the format spec. The serializer cannot delegate numbers +to `JSON.stringify` (it loses `-0` and non-finite values); DataJS owns its +number writer. Whether the canonical layout is fully minified or one statement +per line is decided in the spec stage. + +**Extensions.** Recognized: `.data.js`, `.data.mjs`, `.d.js`, `.d.mjs`. +Emitted and canonical: `.data.js` (`.data.mjs` where unambiguous ESM +resolution matters). No `.f` combinations — `.f.[m]js` marks FunctionalScript +source, and every DataJS document is compiler-accepted by construction, so a +combined marker would encode a redundant fact. + +### FunctionalScript consequences + +- **`;` is required in early-stage FunctionalScript**, matching DataJS. This + removes ASI — including its future "no LineTerminator here" restricted + productions — before the expression grammar grows the hazards (`(`, `[` at + line start). Relaxing later to also accept newline termination is + backward-compatible; the reverse would be breaking, so strict-first is the + safe ratchet. Repository `.f.mjs` source is unaffected (it is parsed by + Node/TypeScript); the cost lands at `.f.mjs` → `.f.js` migration, where the + normalizer inserts `;` mechanically — `.f.js` is compiler-formatted, not + hand-formatted. +- **`undefined`, `NaN`, `Infinity` become FunctionalScript reserved words**, + so the DataJS binding restriction is inherited rather than special-cased. +- The moved parser's separator rule changes from newline to `';'` (the moved + tokenizer's operator vocabulary gains `;`). +- Subset laws are proof obligations, not prose: every DataJS *accept* vector + parses in FunctionalScript to the same value graph; the normalizer closes + the loop (`parse_datajs(normalize(m))` equals the evaluation of any + data-only module `m`); FunctionalScript fixtures remain valid JS with + identical meaning (checked against a real JS engine in proofs). + +### Stages + +Each stage lands green and independently; `fjs compile` keeps working +throughout. + +1. **Spec** — `spec/datajs/`: format spec (grammar as BNF text, data model, + rationale) plus the normalization section, and the conformance test + vectors (accept, reject, round-trip) that every later stage runs against. + Decides the two deferred details: canonical layout, media type. +2. **Dead code** — delete `fjs/fsc/bnf.f.mjs` and `fjs/fsc/json.f.mjs`, or + convert the salvageable parts into proof-covered `fjs/bnf/**` examples. + Resolves [orphaned-json-grammar](../fjs/fsc/todo/orphaned-json-grammar.md). +3. **JSON self-contained tokenizer** — replace the `fjs/js/tokenizer` wrapper + in `fjs/media/json/tokenizer` with a scanner of JSON's own lexical + grammar, exporting the string and number scanners for reuse. + Accepted-input proofs unchanged; error-shape proofs rewritten once. +4. **`fjs/media/datajs`** — parser (JSON's container machine via its policy + seam, plus an identifier policy) and serializer (the shared walker of + [157](../fjs/djs/todo/157-json-djs-shared-value-machine.md) §2 with a + ref-lookup hook, own number writer), proofs over the spec vectors. +5. **Front-end move** — `fjs/djs/{tokenizer,parser,ast,transpiler}` → + `fjs/fsc/*` as a rename; separator `nl` → `';'`; reserved words added; + `fjs compile` repointed. The EDAG staging + ([compile-modules-to-edag](../fjs/djs/todo/compile-modules-to-edag.md)) + continues under the `fsc` name. +6. **Compiler output** — the normalizer: data-only FunctionalScript (imports + resolved and inlined) to normalized DataJS or JSON, with the subset-law + proofs above. +7. **Cleanup** — retire `fjs/js/tokenizer` when its last consumer is gone + (`fjs/js/string_escape` and `fjs/js/keywords` remain as shared, + JS-spec-frozen tables); one clean-break release with the standard + `**BREAKING CHANGES:**` changelog treatment for the removed `fjs/djs/*` + paths and changed serializer output — no compatibility shims. + +### Tasks + +- [ ] Stage 1: write `spec/datajs/` and the conformance vectors; file its + co-located todo. +- [ ] Stage 2: resolve + [orphaned-json-grammar](../fjs/fsc/todo/orphaned-json-grammar.md). +- [ ] Stage 3: JSON self-contained tokenizer; file its todo under + `fjs/media/json/todo/`. +- [ ] Stage 4: `fjs/media/datajs`; file its todo. +- [ ] Stage 5: front-end move to `fjs/fsc`; file its todo. +- [ ] Stage 6: normalizer + subset-law proofs; file its todo. +- [ ] Stage 7: `fjs/js/tokenizer` retirement and the breaking-change release. +- [ ] Update affected issues as their subject matter moves (see below). +- [ ] `npx tsc`, `fjs test` at every stage. + +### Edits owed to existing issues + +- [157-json-djs-shared-value-machine](../fjs/djs/todo/157-json-djs-shared-value-machine.md) + — §2's shared-walker extraction becomes stage 4 work; §3's minus-rewriter + question is settled by stage 3 (the folding lives in JSON's own tokenizer + and DataJS reuses it). Rebase the issue on this plan or fold it in. +- [663-json-djs-tree-type](../fjs/djs/todo/663-json-djs-tree-type.md) — the + shared `Tree

` instantiation targets `fjs/media/datajs`; rename paths. +- [bnf-grammar-single-owner](../fjs/media/json/todo/bnf-grammar-single-owner.md) + — re-scope: the canonical JSON grammar's owner is the spec (text) plus a + proof-covered `fjs/bnf` example, not a runtime module; the + `fjs/djs/tokenizer` pointer becomes the `fsc` tokenizer. +- [compile-modules-to-edag](../fjs/djs/todo/compile-modules-to-edag.md) — its + paths move `djs` → `fsc` in stage 5; its special-number round-trip + requirement is satisfied by the DataJS spec rather than DJS-specific + patches. +- [orphaned-json-grammar](../fjs/fsc/todo/orphaned-json-grammar.md) — + resolved by stage 2. +- `fjs/djs/README.md` and the remaining `fjs/djs/todo/*` files — move with + their subject matter in stage 5; the DJS name in them refers to the moved + front end, not to DataJS. + +### Related + +- [`fjs/media/json/README.md`](../fjs/media/json/README.md) — the policy-seam + parser design DataJS layers on. +- [`fjs/djs/parser/README.md`](../fjs/djs/parser/README.md) — the front end + that moves to `fjs/fsc`. +- [`todo/edag-stage1-discussion.md`](./edag-stage1-discussion.md), + [`todo/edag-spec.md`](./edag-spec.md) — EDAG semantics the moved front end + compiles to; serialized EDAG spells object constructors as arrays, so + DataJS's JS-derived object semantics do not conflict with EDAG's ordered + entries. +- [`fjs/fsc/README.md`](../fjs/fsc/README.md) — the `.f.mjs` → `.f.js` + migration where the `;` requirement lands. +- [`todo/migrate-typescript-to-mjs.md`](./migrate-typescript-to-mjs.md) — the + repository-wide source migration this plan slots into. From 1cbbb0e72004ed0d0c1138602efe53e03ea7f25a Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 28 Aug 2026 21:14:31 +0000 Subject: [PATCH 02/14] todo: apply review findings to the restructure plan Three review findings on the DataJS decision record, each a subset-law hole: const names must also exclude JavaScript's reserved words in module code (const class = 1 is a JS syntax error); the JSON-to-DataJS textual conversion must rewrite a bare "__proto__" key to the computed spelling rather than claiming plain concatenation is always valid; and bigint is its own digits-only production, not an 'n' suffix on the JSON number grammar (JS rejects 1.5n and 1e2n). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho --- todo/parser-serializer-restructure.md | 33 +++++++++++++++++++-------- 1 file changed, 23 insertions(+), 10 deletions(-) diff --git a/todo/parser-serializer-restructure.md b/todo/parser-serializer-restructure.md index 4f656c20f..5cdf7e4bc 100644 --- a/todo/parser-serializer-restructure.md +++ b/todo/parser-serializer-restructure.md @@ -121,20 +121,33 @@ key ::= string | '[' '"__proto__"' ']' Other JS whitespace (U+2028/U+2029, NBSP, FF, BOM) is rejected. - **No comments, no imports.** A DataJS document is closed; the compiler inlines resolved imports when normalizing FunctionalScript to DataJS. -- **Strings and numbers are JSON's grammar**, plus the bigint `n` suffix. - `-` is not an operator: it folds into a following number, bigint, or - `Infinity` token only (`-NaN`, `-undefined`, a bare `-` are rejected). +- **Strings and numbers are JSON's grammar.** Bigint is a production of its + own, not a suffix on the number grammar: JSON's integer part (no fraction, + no exponent, no leading zeros) followed by `n` — JS rejects `1.5n` and + `1e2n`, so "number + `n`" would over-accept. `-` is not an operator: it + folds into a following number, bigint, or `Infinity` token only (`-NaN`, + `-undefined`, a bare `-` are rejected). - **Keys** are JSON strings, plus the computed spelling `["__proto__"]` as the only way to write that one key; a bare or string `"__proto__"` key is rejected (JS would read it as prototype replacement). - **Const names** are ASCII: `[A-Za-z_$][A-Za-z0-9_$]*`, each bound once, - and binding `undefined`, `NaN`, or `Infinity` is rejected — JS permits - `const undefined = 5` and later `undefined` then means the const, which a - subset treating it as a literal would silently reinterpret. -- **A JSON document is a valid DataJS value, never a DataJS document** (a - DataJS document is a JS module, so it cannot be a JSON document). The - conversion is literal: `"export default " + json + ";"` — minus the `;`, - `"export default " + json` — is always a valid document. + minus two exclusion sets. JavaScript's reserved words as they apply to a + binding identifier in module code (module code is strict), including + `import`, `export`, `let`, `yield`, `await`, and `static`, are excluded — + `const class = 1` is a JS syntax error, so accepting it would break the + subset law. Binding `undefined`, `NaN`, or `Infinity` is additionally + rejected — JS *permits* `const undefined = 5` and later `undefined` then + means the const, which a subset treating it as a literal would silently + reinterpret. The spec enumerates the excluded words exhaustively rather + than citing ECMA-262. +- **Every JSON value is a DataJS value; no JSON document is a DataJS + document** (a DataJS document is a JS module, so it cannot be a JSON + document). The textual conversion `"export default " + json` yields a + valid document with one exception: a bare `"__proto__"` object key — + rejected by DataJS because JS reads it as prototype replacement — must be + rewritten to the computed spelling `["__proto__"]` during conversion. + Plain concatenation is exactly valid for JSON containing no `__proto__` + key. **Serialization.** Any conforming serializer may emit any valid document; a separate *normalized form* section defines one byte-deterministic canonical From 083f6cf9e016a28d7e91c09287c5de7248ead21d Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 28 Aug 2026 21:17:24 +0000 Subject: [PATCH 03/14] todo: const names also exclude strict-mode eval and arguments Review follow-up: eval and arguments are not reserved words, but module code is strict and rejects binding them, so the exclusion set is "every name JS rejects as a binding identifier in module code", not "reserved words" alone. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho --- todo/parser-serializer-restructure.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/todo/parser-serializer-restructure.md b/todo/parser-serializer-restructure.md index 5cdf7e4bc..34f2e353e 100644 --- a/todo/parser-serializer-restructure.md +++ b/todo/parser-serializer-restructure.md @@ -131,11 +131,13 @@ key ::= string | '[' '"__proto__"' ']' only way to write that one key; a bare or string `"__proto__"` key is rejected (JS would read it as prototype replacement). - **Const names** are ASCII: `[A-Za-z_$][A-Za-z0-9_$]*`, each bound once, - minus two exclusion sets. JavaScript's reserved words as they apply to a - binding identifier in module code (module code is strict), including - `import`, `export`, `let`, `yield`, `await`, and `static`, are excluded — - `const class = 1` is a JS syntax error, so accepting it would break the - subset law. Binding `undefined`, `NaN`, or `Infinity` is additionally + minus two exclusion sets. Every name JavaScript rejects as a binding + identifier in module code (module code is strict) is excluded: the + reserved words, including `import`, `export`, `let`, `yield`, `await`, + and `static`, and the strict-mode-only bindings `eval` and `arguments` — + `const class = 1` and `const eval = 1` are JS syntax errors there, so + accepting either would break the subset law. Binding `undefined`, `NaN`, + or `Infinity` is additionally rejected — JS *permits* `const undefined = 5` and later `undefined` then means the const, which a subset treating it as a literal would silently reinterpret. The spec enumerates the excluded words exhaustively rather From b33029dc3f244f137ba2f60ce5c297cc0d0debd1 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 28 Aug 2026 21:22:48 +0000 Subject: [PATCH 04/14] todo: const hoisting by identity, stage-5 numeric leaves and changelog Three review findings: normalized-form const hoisting is restricted to objects/arrays counted by reference identity, so a value-equality ref counter can never merge 0 with -0 or mishandle NaN; stage 5 gains the front-end work for NaN/Infinity/-Infinity/-0 (unresolved identifiers in today's parser), without which the stage-6 subset proofs reject DataJS accept vectors; and stage 5's syntax change carries its own BREAKING CHANGES changelog entry rather than deferring it to stage 7. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho --- todo/parser-serializer-restructure.md | 29 ++++++++++++++++++++------- 1 file changed, 22 insertions(+), 7 deletions(-) diff --git a/todo/parser-serializer-restructure.md b/todo/parser-serializer-restructure.md index 34f2e353e..5d3134223 100644 --- a/todo/parser-serializer-restructure.md +++ b/todo/parser-serializer-restructure.md @@ -154,7 +154,11 @@ key ::= string | '[' '"__proto__"' ']' **Serialization.** Any conforming serializer may emit any valid document; a separate *normalized form* section defines one byte-deterministic canonical serializer (const names `_0`, `_1`, … in first-emission order; a const emitted -iff its value is referenced more than once; shortest round-trip number +iff its value is an object or array referenced more than once **by reference +identity** — primitives are always emitted inline and never hoisted, since +primitive sharing is unobservable and a value-equality ref counter would +face the `0`/`-0` and `NaN` merging ambiguity that the `Object.is` +round-trip guarantee forbids; shortest round-trip number spelling; bigints as full digits + `n`; fixed string escaping). Normalization is not a blocker for the format spec. The serializer cannot delegate numbers to `JSON.stringify` (it loses `-0` and non-finite values); DataJS owns its @@ -210,17 +214,28 @@ throughout. ref-lookup hook, own number writer), proofs over the spec vectors. 5. **Front-end move** — `fjs/djs/{tokenizer,parser,ast,transpiler}` → `fjs/fsc/*` as a rename; separator `nl` → `';'`; reserved words added; - `fjs compile` repointed. The EDAG staging - ([compile-modules-to-edag](../fjs/djs/todo/compile-modules-to-edag.md)) - continues under the `fsc` name. + the DataJS numeric leaves taught to the moved front end — `NaN`, + `Infinity`, `-Infinity`, and exact `-0` are unresolved identifiers in + today's parser, so reserving the names alone would *reject* DataJS accept + vectors: tokenizer, grammar, minus-folding, and AST/evaluation support is + stage-5 work (the front-end half of + [compile-modules-to-edag](../fjs/djs/todo/compile-modules-to-edag.md)'s + special-number requirement), a precondition of stage 6's subset proofs; + `fjs compile` repointed. The EDAG staging continues under the `fsc` + name. This stage changes accepted public `.f.js` syntax (statement + termination, newly reserved names), so its own PR carries the + `**BREAKING CHANGES:**` changelog treatment for that behavior — it is + not deferred to stage 7. 6. **Compiler output** — the normalizer: data-only FunctionalScript (imports resolved and inlined) to normalized DataJS or JSON, with the subset-law proofs above. 7. **Cleanup** — retire `fjs/js/tokenizer` when its last consumer is gone (`fjs/js/string_escape` and `fjs/js/keywords` remain as shared, - JS-spec-frozen tables); one clean-break release with the standard - `**BREAKING CHANGES:**` changelog treatment for the removed `fjs/djs/*` - paths and changed serializer output — no compatibility shims. + JS-spec-frozen tables); the clean-break release with `**BREAKING + CHANGES:**` changelog treatment for the removed `fjs/djs/*` paths and + changed serializer output — no compatibility shims. (Each earlier stage + that changes public behavior, stage 5 in particular, carries its own + breaking-change entry in its own PR, per the changelog convention.) ### Tasks From 4182601cc196022a8a3d20b95ca770e6b2fc47dd Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 28 Aug 2026 21:23:53 +0000 Subject: [PATCH 05/14] todo: name every fjs/djs destination; EOF change already shipped Review follow-ups from the human approval: stage 5 now states where the rest of fjs/djs lands (serializer and value-tree types into stage 4's fjs/media/datajs, examples and the top-level compile() module with the front end to fsc), and the compile-modules-to-edag edit note distinguishes its front-end paths (djs -> fsc) from its serializer citation (-> media/ datajs). The bnf EOF-encoding change is cited as shipped (#1516) rather than listed as still pending. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho --- todo/parser-serializer-restructure.md | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/todo/parser-serializer-restructure.md b/todo/parser-serializer-restructure.md index 5d3134223..0f623c0e0 100644 --- a/todo/parser-serializer-restructure.md +++ b/todo/parser-serializer-restructure.md @@ -25,8 +25,10 @@ relationships grew rather than being designed: - **`fjs/fsc`** — nearly empty: a character-classifier stub plus a dead third copy of the JSON grammar ([orphaned-json-grammar](../fjs/fsc/todo/orphaned-json-grammar.md)). -- **`fjs/bnf`** — the grammar toolkit, still evolving (EOF encoding change, - pending unicode split). +- **`fjs/bnf`** — the grammar toolkit, still evolving: a breaking + EOF-encoding change shipped recently + ([#1516](https://github.com/functionalscript/functionalscript/pull/1516)), + and the unicode split is still pending. Two structural problems follow: @@ -213,7 +215,13 @@ throughout. [157](../fjs/djs/todo/157-json-djs-shared-value-machine.md) §2 with a ref-lookup hook, own number writer), proofs over the spec vectors. 5. **Front-end move** — `fjs/djs/{tokenizer,parser,ast,transpiler}` → - `fjs/fsc/*` as a rename; separator `nl` → `';'`; reserved words added; + `fjs/fsc/*` as a rename. The rest of `fjs/djs` has stated destinations + rather than following the rename: `serializer/` is reworked into stage + 4's `fjs/media/datajs` (it does not move to `fsc`); the value-tree types + in `fjs/djs/types.ts` go with it, per + [663](../fjs/djs/todo/663-json-djs-tree-type.md); `examples/` and the + top-level `module.f.mjs`/`proof.f.mjs` carrying `compile()` move with + the front end to `fsc`. Separator `nl` → `';'`; reserved words added; the DataJS numeric leaves taught to the moved front end — `NaN`, `Infinity`, `-Infinity`, and exact `-0` are unresolved identifiers in today's parser, so reserving the names alone would *reject* DataJS accept @@ -265,7 +273,9 @@ throughout. proof-covered `fjs/bnf` example, not a runtime module; the `fjs/djs/tokenizer` pointer becomes the `fsc` tokenizer. - [compile-modules-to-edag](../fjs/djs/todo/compile-modules-to-edag.md) — its - paths move `djs` → `fsc` in stage 5; its special-number round-trip + front-end paths move `djs` → `fsc` in stage 5, while its serializer + citation (`../serializer/module.f.mjs`) follows the serializer into + stage 4's `fjs/media/datajs`; its special-number round-trip requirement is satisfied by the DataJS spec rather than DJS-specific patches. - [orphaned-json-grammar](../fjs/fsc/todo/orphaned-json-grammar.md) — From 91eb2d2974cc93df2b465b7262aaa43b84b93c6e Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 28 Aug 2026 21:29:02 +0000 Subject: [PATCH 06/14] todo: stage 6 JSON output is rejected when unrepresentable Review finding: the normalizer's JSON output needs a representability rule. DataJS output is total; JSON output is permitted only when every leaf has a JSON spelling and no graph sharing is lost, and otherwise the value is rejected as an error rather than substituted or dropped, matching json-bigint-serialization's validation policy. Rejection proofs cover each unrepresentable leaf and the shared-node case. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho --- todo/parser-serializer-restructure.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/todo/parser-serializer-restructure.md b/todo/parser-serializer-restructure.md index 0f623c0e0..2218b7167 100644 --- a/todo/parser-serializer-restructure.md +++ b/todo/parser-serializer-restructure.md @@ -236,7 +236,15 @@ throughout. not deferred to stage 7. 6. **Compiler output** — the normalizer: data-only FunctionalScript (imports resolved and inlined) to normalized DataJS or JSON, with the subset-law - proofs above. + proofs above. DataJS output is total; JSON output is permitted only when + every leaf has a JSON spelling and no graph sharing is lost — a value + containing `undefined`, `NaN`, `±Infinity`, or a shared node is + **rejected as an error**, never silently substituted or dropped, + matching the validation policy of + [json-bigint-serialization](../fjs/djs/todo/json-bigint-serialization.md) + (`bigint` itself is representable: it serializes as its full digits). + Rejection proofs cover each unrepresentable leaf and the shared-node + case. 7. **Cleanup** — retire `fjs/js/tokenizer` when its last consumer is gone (`fjs/js/string_escape` and `fjs/js/keywords` remain as shared, JS-spec-frozen tables); the clean-break release with `**BREAKING From 697537614205e45ef81661ec2f68b2cb7da2b40e Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 28 Aug 2026 21:30:57 +0000 Subject: [PATCH 07/14] todo: canonical DataJS layout is one line; readable output is the default Resolves the deferred canonical-layout decision: normalized form is the fully minified one-line spelling, leaving normalization zero layout freedom for byte-determinism; tooling defaults to a human-readable layout, which is one of the many valid non-normalized spellings. The media type remains the spec stage's one deferred detail. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho --- todo/parser-serializer-restructure.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/todo/parser-serializer-restructure.md b/todo/parser-serializer-restructure.md index 2218b7167..c3d33cd05 100644 --- a/todo/parser-serializer-restructure.md +++ b/todo/parser-serializer-restructure.md @@ -164,8 +164,12 @@ round-trip guarantee forbids; shortest round-trip number spelling; bigints as full digits + `n`; fixed string escaping). Normalization is not a blocker for the format spec. The serializer cannot delegate numbers to `JSON.stringify` (it loses `-0` and non-finite values); DataJS owns its -number writer. Whether the canonical layout is fully minified or one statement -per line is decided in the spec stage. +number writer. The canonical layout is **one line** — fully minified, with +whitespace only where two word-tokens meet — so normalization has zero +layout freedom, which is what byte-determinism (and any future content +addressing) needs. Tooling *defaults* to a human-readable layout (one +statement per line, indented containers), which is simply one of the many +valid non-normalized spellings; normalized output is requested explicitly. **Extensions.** Recognized: `.data.js`, `.data.mjs`, `.d.js`, `.d.mjs`. Emitted and canonical: `.data.js` (`.data.mjs` where unambiguous ESM @@ -202,7 +206,8 @@ throughout. 1. **Spec** — `spec/datajs/`: format spec (grammar as BNF text, data model, rationale) plus the normalization section, and the conformance test vectors (accept, reject, round-trip) that every later stage runs against. - Decides the two deferred details: canonical layout, media type. + Decides the one remaining deferred detail: the media type. (The + canonical layout is decided: one line — see **Serialization** above.) 2. **Dead code** — delete `fjs/fsc/bnf.f.mjs` and `fjs/fsc/json.f.mjs`, or convert the salvageable parts into proof-covered `fjs/bnf/**` examples. Resolves [orphaned-json-grammar](../fjs/fsc/todo/orphaned-json-grammar.md). From fd68a5542a82068d9cc5749e4491f8d3940193ee Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 28 Aug 2026 21:35:36 +0000 Subject: [PATCH 08/14] todo: object key order is JS own-property order, integer keys included Review finding: "position from the first occurrence" alone lets a non-JS implementation preserve {"2":0,"1":0} as written, while every JS engine observably enumerates "1" before "2" - array-index keys come first in ascending numeric order, then other keys in first-occurrence order. The data model now states JS own-property ordering explicitly and normalized output emits keys in that observable order. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho --- todo/parser-serializer-restructure.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/todo/parser-serializer-restructure.md b/todo/parser-serializer-restructure.md index c3d33cd05..5794fc62d 100644 --- a/todo/parser-serializer-restructure.md +++ b/todo/parser-serializer-restructure.md @@ -92,8 +92,15 @@ package publishes; the spec does not need it. **Data model.** A DAG of values. Leaves are JSON's primitives plus `bigint`, `undefined`, `NaN`, `Infinity`, `-Infinity`, `-0`. Number round-trips satisfy -`Object.is`. Object entries follow JS duplicate-key semantics exactly: value -from the last occurrence, position from the first. Sharing is semantic — two +`Object.is`. Object entries follow JS object semantics exactly, and the spec +restates both halves rather than citing ECMA-262. Duplicate keys: value from +the last occurrence, position from the first. Observable key order is JS's +own-property ordering: keys that are array indices (canonical numeric +strings, `0` ≤ n < 2^32−1) come first in ascending numeric order, then all +other keys in first-occurrence order — `{"2":0,"1":0}` observably orders +`"1"` before `"2"` in every JS engine, and a non-JS implementation must +reorder the same way. Normalized output emits keys in that observable +order. Sharing is semantic — two references to one `const` denote the same node, and references may only point at *earlier* consts, so a document is acyclic by construction and parseable in one pass. The reference parser returns live JS values and does not freeze them From 3015f9d232cb9795f4df9011f743f2c523d78d78 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 28 Aug 2026 21:41:29 +0000 Subject: [PATCH 09/14] todo: post-order const emission; JSON output rejects bigint Two review findings: "first-emission order" was underdetermined for nested shared nodes, so normalization now emits consts in post-order of one depth-first traversal (arrays in element order, objects in observable key order, shared nodes descended on first encounter), which also makes declaration-before-use automatic; and the normalizer's ordinary JSON output rejects bigint alongside the other unrepresentable leaves, since emitting 1n as the JSON text 1 silently changes the value's type for the standard reader - extended-codec output stays an explicit, labeled caller choice. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho --- todo/parser-serializer-restructure.md | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/todo/parser-serializer-restructure.md b/todo/parser-serializer-restructure.md index 5794fc62d..1df813128 100644 --- a/todo/parser-serializer-restructure.md +++ b/todo/parser-serializer-restructure.md @@ -162,9 +162,16 @@ key ::= string | '[' '"__proto__"' ']' **Serialization.** Any conforming serializer may emit any valid document; a separate *normalized form* section defines one byte-deterministic canonical -serializer (const names `_0`, `_1`, … in first-emission order; a const emitted +serializer (a const emitted iff its value is an object or array referenced more than once **by reference -identity** — primitives are always emitted inline and never hoisted, since +identity**; consts are emitted in **post-order of one depth-first traversal** +of the root value — arrays in element order, objects in observable key +order, each shared node descended into only on first encounter — with names +`_0`, `_1`, … assigned in emission order, so a shared node's dependencies +are always declared before it and "who is `_0`" has exactly one answer: +for `root = [parent, parent, child]` with `child` inside `parent`, `child` +finishes first and is `_0`, `parent` is `_1`; primitives are always emitted +inline and never hoisted, since primitive sharing is unobservable and a value-equality ref counter would face the `0`/`-0` and `NaN` merging ambiguity that the `Object.is` round-trip guarantee forbids; shortest round-trip number @@ -250,11 +257,15 @@ throughout. resolved and inlined) to normalized DataJS or JSON, with the subset-law proofs above. DataJS output is total; JSON output is permitted only when every leaf has a JSON spelling and no graph sharing is lost — a value - containing `undefined`, `NaN`, `±Infinity`, or a shared node is - **rejected as an error**, never silently substituted or dropped, + containing `undefined`, `NaN`, `±Infinity`, `bigint`, or a shared node + is **rejected as an error**, never silently substituted or dropped, matching the validation policy of - [json-bigint-serialization](../fjs/djs/todo/json-bigint-serialization.md) - (`bigint` itself is representable: it serializes as its full digits). + [json-bigint-serialization](../fjs/djs/todo/json-bigint-serialization.md). + `bigint` is rejected even though its digits are spellable in JSON: the + text `1` read back by the standard `.json` reader is the *number* `1`, + so emitting `1n` as `1` would silently change the value's type — the + extended codec's bigint output remains available only as a caller's + explicit, so-labeled choice, never the normalizer's `.json` default. Rejection proofs cover each unrepresentable leaf and the shared-node case. 7. **Cleanup** — retire `fjs/js/tokenizer` when its last consumer is gone From fd4d6d2dcc022d907d8dfd019c0e6bfd9725fa5d Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 28 Aug 2026 21:45:30 +0000 Subject: [PATCH 10/14] todo: canonical numbers are ToString(Number); serializer rejects cycles Two review findings: "shortest round-trip spelling" does not select unique bytes (1e3 vs 1E3), so the canonical number spelling is exactly ECMAScript's deterministic ToString(Number) algorithm, restated in the spec, with -0 as the one stated exception (ToString spells it 0, canonical DataJS spells it -0); and the serializer's input is a live unfrozen value that may be cyclic, so cycle detection and a proved rejection are required - a DataJS document can only represent a DAG. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho --- todo/parser-serializer-restructure.md | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/todo/parser-serializer-restructure.md b/todo/parser-serializer-restructure.md index 1df813128..e20b9b106 100644 --- a/todo/parser-serializer-restructure.md +++ b/todo/parser-serializer-restructure.md @@ -174,8 +174,17 @@ finishes first and is `_0`, `parent` is `_1`; primitives are always emitted inline and never hoisted, since primitive sharing is unobservable and a value-equality ref counter would face the `0`/`-0` and `NaN` merging ambiguity that the `Object.is` -round-trip guarantee forbids; shortest round-trip number -spelling; bigints as full digits + `n`; fixed string escaping). Normalization +round-trip guarantee forbids; the canonical number spelling is exactly +ECMAScript's `ToString(Number)` — a fully deterministic algorithm the spec +restates, so no "shortest spelling" tie such as `1e3` vs `1E3` exists, +`ToString` never produces the uppercase form — with one stated exception, +`-0`, which `ToString` spells `0` and canonical DataJS spells `-0`; +bigints as full digits + `n`; fixed string escaping). The serializer's +*input* is a programmatic value that is not frozen and may be cyclic +(`value.self = value`); DataJS represents DAGs only, so the serializer +detects cycles and rejects them as an error — never emitting a +self-referencing `const _0={"self":_0};` (a TDZ failure in JS) and never +recursing unboundedly — with rejection proofs in stage 4. Normalization is not a blocker for the format spec. The serializer cannot delegate numbers to `JSON.stringify` (it loses `-0` and non-finite values); DataJS owns its number writer. The canonical layout is **one line** — fully minified, with From 0cd6ace67aefca30cdccc3168ed085544c79c7d7 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 28 Aug 2026 21:50:35 +0000 Subject: [PATCH 11/14] todo: every statement ends with ';', export default included Design change: one uniform terminator rule instead of a rule plus an export-default exception - simpler to spec and implement, still a JS subset since export default value; is valid JS. The grammar, the one-line example, and the JSON-to-DataJS conversion (now "export default " + json + ";") follow. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho --- todo/parser-serializer-restructure.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/todo/parser-serializer-restructure.md b/todo/parser-serializer-restructure.md index e20b9b106..54171b4d8 100644 --- a/todo/parser-serializer-restructure.md +++ b/todo/parser-serializer-restructure.md @@ -112,18 +112,19 @@ other implementations may. ```text module ::= const* export const ::= 'const' id '=' value ';' -export ::= 'export' 'default' value (no trailing ';') +export ::= 'export' 'default' value ';' value ::= primitive | id | array | object key ::= string | '[' '"__proto__"' ']' ``` -- **`;` terminates every `const`;** no `;` after `export default`, no empty +- **`;` terminates every statement, `export default` included** — one + uniform rule, no per-statement exception; no empty statements. Rationale, each sufficient alone: no line-terminator taxonomy in the spec (a lone CR *is* a JS `LineTerminator` — trivia no implementer should need); one canonical spelling per document; the separator is a visible character, so byte-different files that render identically cannot differ in meaning; and a document minifies to one line — - `const a=[];export default[a,a]` — enabling DataJS inside JSON strings, + `const a=[];export default[a,a];` — enabling DataJS inside JSON strings, line-delimited streaming, and one-line test fixtures. Whitespace is needed only between adjacent word-tokens (`const a`, `export default x`). - **Whitespace is JSON's** — space, tab, LF, CR — insignificant everywhere. @@ -153,7 +154,7 @@ key ::= string | '[' '"__proto__"' ']' than citing ECMA-262. - **Every JSON value is a DataJS value; no JSON document is a DataJS document** (a DataJS document is a JS module, so it cannot be a JSON - document). The textual conversion `"export default " + json` yields a + document). The textual conversion `"export default " + json + ";"` yields a valid document with one exception: a bare `"__proto__"` object key — rejected by DataJS because JS reads it as prototype replacement — must be rewritten to the computed spelling `["__proto__"]` during conversion. From 1ee9e89e3ede6c557cb9d3284455c8ca690b8ce8 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 28 Aug 2026 21:51:29 +0000 Subject: [PATCH 12/14] todo: serializer validates the whole data model; -0 already parses Two review findings: serializer input validation generalizes beyond cycles - any value outside the DataJS data model is rejected rather than approximated (foreign leaves, sparse-array holes, symbol-keyed and accessor properties, cycles), each with a rejection proof; and stage 5 no longer directs reimplementing exact -0, which the current front end already parses correctly (lexeme pinned in the tokenizer proof, parseFloat preserves signed zero) - it gets a regression proof instead. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho --- todo/parser-serializer-restructure.md | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/todo/parser-serializer-restructure.md b/todo/parser-serializer-restructure.md index 54171b4d8..7438fac72 100644 --- a/todo/parser-serializer-restructure.md +++ b/todo/parser-serializer-restructure.md @@ -181,11 +181,15 @@ restates, so no "shortest spelling" tie such as `1e3` vs `1E3` exists, `ToString` never produces the uppercase form — with one stated exception, `-0`, which `ToString` spells `0` and canonical DataJS spells `-0`; bigints as full digits + `n`; fixed string escaping). The serializer's -*input* is a programmatic value that is not frozen and may be cyclic -(`value.self = value`); DataJS represents DAGs only, so the serializer -detects cycles and rejects them as an error — never emitting a -self-referencing `const _0={"self":_0};` (a TDZ failure in JS) and never -recursing unboundedly — with rejection proofs in stage 4. Normalization +*input* is a programmatic value that is not frozen, so it must be validated +against the DataJS data model, and anything outside the model is rejected +as an error rather than approximated: a leaf outside the leaf set (a +function, a symbol, a `Date` or any other non-plain object), a sparse +array's hole (which is not an `undefined` element), a symbol-keyed or +accessor own property (reading a getter is an effect), and a cycle +(`value.self = value`) — DataJS represents DAGs only, and treating a +back-edge as sharing would emit a self-referencing `const _0={"self":_0};`, +a TDZ failure in JS. Rejection proofs in stage 4 cover each case. Normalization is not a blocker for the format spec. The serializer cannot delegate numbers to `JSON.stringify` (it loses `-0` and non-finite values); DataJS owns its number writer. The canonical layout is **one line** — fully minified, with @@ -252,10 +256,13 @@ throughout. top-level `module.f.mjs`/`proof.f.mjs` carrying `compile()` move with the front end to `fsc`. Separator `nl` → `';'`; reserved words added; the DataJS numeric leaves taught to the moved front end — `NaN`, - `Infinity`, `-Infinity`, and exact `-0` are unresolved identifiers in + `Infinity`, and `-Infinity` are unresolved identifiers in today's parser, so reserving the names alone would *reject* DataJS accept - vectors: tokenizer, grammar, minus-folding, and AST/evaluation support is - stage-5 work (the front-end half of + vectors: their tokenizer, grammar, minus-folding, and AST/evaluation + support is stage-5 work; exact `-0` already parses correctly (the + tokenizer pins the `-0` lexeme and `parseFloat` preserves signed zero), + so it needs a regression proof, not reimplementation (together the + front-end half of [compile-modules-to-edag](../fjs/djs/todo/compile-modules-to-edag.md)'s special-number requirement), a precondition of stage 6's subset proofs; `fjs compile` repointed. The EDAG staging continues under the `fsc` From 8ff9dd8b894b5dae5c8781176fede4846654190b Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 28 Aug 2026 21:57:13 +0000 Subject: [PATCH 13/14] todo: name the seam generalization; minus folding is parameterized Two review findings on implementation feasibility: stage 4 now states that today's JSON parser seam is too narrow for DataJS (NumberPolicy sees number tokens only, no identifier/bigint tokens, string keys only) and makes generalizing it - token vocabulary, leaf/identifier policy hook, key-form hook, with JSON behavior pinned unchanged by proofs - explicit prerequisite work; and minus folding becomes a parameterized helper whose strict JSON instantiation folds a number only, while DataJS's adds -Infinity and negative bigint, so the extra sign forms never enter the JSON tokenizer. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho --- todo/parser-serializer-restructure.md | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/todo/parser-serializer-restructure.md b/todo/parser-serializer-restructure.md index 7438fac72..98818aa2e 100644 --- a/todo/parser-serializer-restructure.md +++ b/todo/parser-serializer-restructure.md @@ -243,10 +243,18 @@ throughout. in `fjs/media/json/tokenizer` with a scanner of JSON's own lexical grammar, exporting the string and number scanners for reuse. Accepted-input proofs unchanged; error-shape proofs rewritten once. -4. **`fjs/media/datajs`** — parser (JSON's container machine via its policy - seam, plus an identifier policy) and serializer (the shared walker of +4. **`fjs/media/datajs`** — parser and serializer, proofs over the spec + vectors. The parser reuses JSON's container machine, and today's seam is + **not wide enough for that**: `NumberPolicy` receives number tokens only, + `JsonToken` has no identifier/bigint/`=` tokens, and the object states + accept string keys only. Generalizing the seam is therefore explicit + stage-4 prerequisite work on `fjs/media/json/parser`: extend the token + vocabulary the machine can be fed, add the leaf/identifier policy hook + (JSON's instantiation: error) and the key-form hook (JSON's: string keys + only), and pin JSON's accepted language and behavior unchanged by proofs + across the API change. The serializer is the shared walker of [157](../fjs/djs/todo/157-json-djs-shared-value-machine.md) §2 with a - ref-lookup hook, own number writer), proofs over the spec vectors. + ref-lookup hook and DataJS's own number writer. 5. **Front-end move** — `fjs/djs/{tokenizer,parser,ast,transpiler}` → `fjs/fsc/*` as a rename. The rest of `fjs/djs` has stated destinations rather than following the rename: `serializer/` is reworked into stage @@ -312,8 +320,11 @@ throughout. - [157-json-djs-shared-value-machine](../fjs/djs/todo/157-json-djs-shared-value-machine.md) — §2's shared-walker extraction becomes stage 4 work; §3's minus-rewriter - question is settled by stage 3 (the folding lives in JSON's own tokenizer - and DataJS reuses it). Rebase the issue on this plan or fold it in. + question is settled by stages 3–4: the folding is a parameterized helper + whose strict JSON instantiation folds `-` before a number token only + (JSON's acceptance unchanged), while DataJS's instantiation adds its own + cases (`-Infinity`, negative bigint) — the extra sign forms never enter + the JSON tokenizer. Rebase the issue on this plan or fold it in. - [663-json-djs-tree-type](../fjs/djs/todo/663-json-djs-tree-type.md) — the shared `Tree

` instantiation targets `fjs/media/datajs`; rename paths. - [bnf-grammar-single-owner](../fjs/media/json/todo/bnf-grammar-single-owner.md) From a8b421a8cb5d4c960cdb58877d1440d0b75a2e6c Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 28 Aug 2026 22:00:13 +0000 Subject: [PATCH 14/14] todo: canonical string escaping is QuoteJSONString Review finding: "fixed string escaping" left \n-vs- and slash-escaping divergence open. The canonical spelling is exactly ECMAScript's QuoteJSONString (what JSON.stringify emits for a string), restated in the spec, matching how canonical numbers anchor to ToString(Number). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho --- todo/parser-serializer-restructure.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/todo/parser-serializer-restructure.md b/todo/parser-serializer-restructure.md index 98818aa2e..ff30698c2 100644 --- a/todo/parser-serializer-restructure.md +++ b/todo/parser-serializer-restructure.md @@ -180,7 +180,13 @@ ECMAScript's `ToString(Number)` — a fully deterministic algorithm the spec restates, so no "shortest spelling" tie such as `1e3` vs `1E3` exists, `ToString` never produces the uppercase form — with one stated exception, `-0`, which `ToString` spells `0` and canonical DataJS spells `-0`; -bigints as full digits + `n`; fixed string escaping). The serializer's +bigints as full digits + `n`; canonical string escaping is exactly +ECMAScript's `QuoteJSONString` — what `JSON.stringify` emits for a string: +the minimal escapes `\"` `\\` `\b` `\t` `\n` `\f` `\r`, other control +characters as `\u00`·two lowercase hex digits, unpaired surrogates as +lowercase `\uXXXX`, everything else literal and `/` never escaped — again a +deterministic algorithm the spec restates rather than a "minimal escaping" +adjective). The serializer's *input* is a programmatic value that is not frozen, so it must be validated against the DataJS data model, and anything outside the model is rejected as an error rather than approximated: a leaf outside the leaf set (a