Add DataJS parser/serializer restructuring design document - #1764
Conversation
A coordinating issue for restructuring the parser/serializer stack into
three tiers: a self-contained JSON codec, a new spec'd DataJS interchange
format (JSON extended from tree to DAG, nothing else) in fjs/media/datajs,
and the current fjs/djs front end moving to fjs/fsc to grow with the
language. Records the design decision log (';'-terminated consts, JSON
whitespace, JS-derived duplicate-key semantics, special-number round-trips,
ASCII const names, subset laws DataJS < FJS < JS), the staged migration
sequence, and the edits owed to the five existing issues it supersedes or
rebases.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
functionalscript | a8b421a | Commit Preview URL Branch Preview URL |
Aug 28 2026, 10:01 PM |
|
You have reached your Codex usage limits for security reviews. Please try again later. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7f33189618
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Three review findings on the DataJS decision record, each a subset-law hole: const names must also exclude JavaScript's reserved words in module code (const class = 1 is a JS syntax error); the JSON-to-DataJS textual conversion must rewrite a bare "__proto__" key to the computed spelling rather than claiming plain concatenation is always valid; and bigint is its own digits-only production, not an 'n' suffix on the JSON number grammar (JS rejects 1.5n and 1e2n). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1cbbb0e720
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Review follow-up: eval and arguments are not reserved words, but module code is strict and rejects binding them, so the exclusion set is "every name JS rejects as a binding identifier in module code", not "reserved words" alone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 083f6cf9e0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
o2alexanderfedin
left a comment
o2alexanderfedin
left a comment
There was a problem hiding this comment.
Approved. Checked the claims against the tree and against a JS engine rather than reading for plausibility, since a design note gets implemented later on trust.
Verified against the code: the fjs/media/json tokenizer really is a ~100-line adapter over fjs/js/tokenizer's 747; the NumberPolicy seam is in that README verbatim; fjs/djs's pipeline and the compile command wire up as described; fjs/fsc is a classifier plus an orphaned third JSON grammar, which its own todo independently confirms; and no exports map exists, which you account for. Nothing in the plan violates the .f.mjs rules. All links resolve, P2/open are valid, no changelog needed.
The newest commits' claims hold too, and I ran them: 1.5n and 1e2n are both SyntaxError, so "number + n" really would over-accept and making bigint its own production is right; 01n is rejected, matching the no-leading-zeros wording. const eval = 1 in strict mode is a SyntaxError while const undefined = 5 is accepted — so the two exclusion sets are genuinely different in kind, exactly as the note now says. That last one was on my could-not-verify list before; it checks out.
One gap still open, cheap to close now:
Stage 5 never says where fjs/djs/serializer/ lands — nor examples/, nor the top-level module.f.mjs/proof.f.mjs/types.ts carrying compile(). It is inferable (serializer into stage 4's fjs/media/datajs, compile() into stage 6's normalizer) but unstated, and it becomes a wrong instruction later: "Edits owed to existing issues" tells compile-modules-to-edag.md that "its paths move djs → fsc in stage 5", while that todo's cited path is ../serializer/module.f.mjs, which by your own staging goes to media/datajs. Your next clause already says as much. One line naming the serializer's destination fixes both.
Minor: line 28 still lists "EOF encoding change" among what is still evolving, but EOF = -1 shipped in #1516 on 2026-08-13, some fifty fjs/bnf commits back.
Three review findings: normalized-form const hoisting is restricted to objects/arrays counted by reference identity, so a value-equality ref counter can never merge 0 with -0 or mishandle NaN; stage 5 gains the front-end work for NaN/Infinity/-Infinity/-0 (unresolved identifiers in today's parser), without which the stage-6 subset proofs reject DataJS accept vectors; and stage 5's syntax change carries its own BREAKING CHANGES changelog entry rather than deferring it to stage 7. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho
Review follow-ups from the human approval: stage 5 now states where the rest of fjs/djs lands (serializer and value-tree types into stage 4's fjs/media/datajs, examples and the top-level compile() module with the front end to fsc), and the compile-modules-to-edag edit note distinguishes its front-end paths (djs -> fsc) from its serializer citation (-> media/ datajs). The bnf EOF-encoding change is cited as shipped (#1516) rather than listed as still pending. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4182601cc1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Review finding: the normalizer's JSON output needs a representability rule. DataJS output is total; JSON output is permitted only when every leaf has a JSON spelling and no graph sharing is lost, and otherwise the value is rejected as an error rather than substituted or dropped, matching json-bigint-serialization's validation policy. Rejection proofs cover each unrepresentable leaf and the shared-node case. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho
…ault Resolves the deferred canonical-layout decision: normalized form is the fully minified one-line spelling, leaving normalization zero layout freedom for byte-determinism; tooling defaults to a human-readable layout, which is one of the many valid non-normalized spellings. The media type remains the spec stage's one deferred detail. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6975376142
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Review finding: "position from the first occurrence" alone lets a non-JS
implementation preserve {"2":0,"1":0} as written, while every JS engine
observably enumerates "1" before "2" - array-index keys come first in
ascending numeric order, then other keys in first-occurrence order. The
data model now states JS own-property ordering explicitly and normalized
output emits keys in that observable order.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fd68a5542a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Two review findings: "first-emission order" was underdetermined for nested shared nodes, so normalization now emits consts in post-order of one depth-first traversal (arrays in element order, objects in observable key order, shared nodes descended on first encounter), which also makes declaration-before-use automatic; and the normalizer's ordinary JSON output rejects bigint alongside the other unrepresentable leaves, since emitting 1n as the JSON text 1 silently changes the value's type for the standard reader - extended-codec output stays an explicit, labeled caller choice. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3015f9d232
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Two review findings: "shortest round-trip spelling" does not select unique bytes (1e3 vs 1E3), so the canonical number spelling is exactly ECMAScript's deterministic ToString(Number) algorithm, restated in the spec, with -0 as the one stated exception (ToString spells it 0, canonical DataJS spells it -0); and the serializer's input is a live unfrozen value that may be cyclic, so cycle detection and a proved rejection are required - a DataJS document can only represent a DAG. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fd4d6d2dcc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Design change: one uniform terminator rule instead of a rule plus an export-default exception - simpler to spec and implement, still a JS subset since export default value; is valid JS. The grammar, the one-line example, and the JSON-to-DataJS conversion (now "export default " + json + ";") follow. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho
Two review findings: serializer input validation generalizes beyond cycles - any value outside the DataJS data model is rejected rather than approximated (foreign leaves, sparse-array holes, symbol-keyed and accessor properties, cycles), each with a rejection proof; and stage 5 no longer directs reimplementing exact -0, which the current front end already parses correctly (lexeme pinned in the tokenizer proof, parseFloat preserves signed zero) - it gets a regression proof instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1ee9e89e3e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Two review findings on implementation feasibility: stage 4 now states that today's JSON parser seam is too narrow for DataJS (NumberPolicy sees number tokens only, no identifier/bigint tokens, string keys only) and makes generalizing it - token vocabulary, leaf/identifier policy hook, key-form hook, with JSON behavior pinned unchanged by proofs - explicit prerequisite work; and minus folding becomes a parameterized helper whose strict JSON instantiation folds a number only, while DataJS's adds -Infinity and negative bigint, so the extra sign forms never enter the JSON tokenizer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8ff9dd8b89
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Review finding: "fixed string escaping" left \n-vs- and slash-escaping divergence open. The canonical spelling is exactly ECMAScript's QuoteJSONString (what JSON.stringify emits for a string), restated in the spec, matching how canonical numbers anchor to ToString(Number). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho
o2alexanderfedin
left a comment
o2alexanderfedin
left a comment
There was a problem hiding this comment.
Approved. The gap I raised is closed — stage 4 now names the serializer explicitly (fjs/media/datajs — parser and serializer), so the instruction to the sibling todo no longer points it at fsc, and the EOF wording is corrected.
The later commits add real content rather than polish: the seam generalization, minus-folding as a parameter, -0 parsing, statement termination, and canonical escaping. I did not re-verify each against a JS engine the way I checked the bigint and const-name claims earlier; those held, which is some evidence for the rest, but not the same thing as having checked them.
|
@codex review |
|
Codex Review: Didn't find any major issues. More of your lovely PRs please. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
3 participants
Summary
This PR adds a comprehensive design document outlining the restructuring of JSON, DataJS, and FunctionalScript parsers and serializers across the codebase. The document serves as a coordinating issue that records design decisions, sequences implementation stages, and maps dependencies between existing issues.
Key Changes
todo/parser-serializer-restructure.md— A detailed design document that:fjs/media/json,fjs/djs,fjs/fsc, andfjs/bnfNotable Details
fjs/bnffjs/js/tokenizer), while the compiler front end moves tofjs/fscand continues evolvinghttps://claude.ai/code/session_01PfpYZSMQKCJvViyebNyLho