Plan generic bigint BNF symbols and JSON value layers - #1455

Summary

Add a layered TODO plan for making BNF alphabet-agnostic, expanding its symbol domain, and introducing an exact intermediate JSON representation:

1. Add an extended JSON parse/serialize layer where bare integer syntax round-trips as bigint, while any numeric token containing . or e / E is always a JavaScript number even when mathematically integral. Exact -0 remains negative-zero number. Bigint serialization always emits full canonical base-10 integer digits and never exponent notation. The JSON numeric tokenizer must become lexeme-first: NumberToken.value is the canonical exact source and token creation must not require eagerly constructing an unbounded bigint coefficient or narrowing an unbounded exponent. Derived numeric data such as bf may be fallible/lazy/optional. The shared structural parser then retains those exact tokens before materializing runtime numbers. Exact schema checks use bounded lexical analysis of the token text rather than huge powers or an unbounded numeric exponent. The extended codec is blocked on explicit policies for exponent overflow, non-finite values, and bare integer tokens beyond the runtime bigint construction limit; valid numeric text must never leak an uncaught runtime conversion failure.
2. Add a standard FunctionalScript JSON codec over the same shared lossless structural parse. The default fjs/media/json.parse / stringify contract is valid, deterministic JSON for the bigint-free json.Unknown domain; it does not have to reproduce host JSON.parse / JSON.stringify edge cases or byte-for-byte number spelling. Standard parsing may materialize json.Unknown directly from the exact NumberToken tree instead of going through ExtendedUnknown, so an oversized integer does not require successful intermediate bigint construction. Standard serialization shares the recursive structural walker but may use its own number formatter.
3. Keep reusable runtime value transforms separate from codec semantics. extendedToStandard recursively converts bigint leaves with Number; standardToExtended uses a simple ordered policy (-0 stays number, Number.isSafeInteger values become bigint, other numbers stay numbers). These utilities do not define parser/stringifier composition and do not depend on native shortest-decimal spelling or other JSON.* compatibility details.
4. Treat exact native JSON.parse / JSON.stringify compatibility as P5 blocked follow-up work. Do not spend P3 time on host-specific spelling, overflow, ordering, or API parity. Once Extended JSON and the standard/extended transforms exist, compatibility can be improved incrementally through documented breaking changes to the default json.* behavior. If we later discover that both contracts are useful, a separate compatible API remains an option. That choice is intentionally deferred until a concrete consumer needs it.
5. Add an RTTI-aware JSON parser over the same token-preserving structural parse rather than first collapsing every decimal/exponent token to a plain extended number. RTTI bigint conversion first uses bounded lexical checks over NumberToken.value. Bare-integer bigint materialization is explicitly fallible: if the runtime cannot construct the requested bigint magnitude, the RTTI parser returns its normal validation/error Result instead of panicking. Decimal/exponent-to-bigint coercion requires exact lexical integrality plus a safe JavaScript-number value before bigint conversion. Arbitrarily long exponent text is compared against counts bounded by input length; it is never narrowed to JavaScript number for exact validation and never drives an enormous 10 ** exponent operation. Thus inputs such as 1.00000000000000001, 1e-99999999999999999999, and oversized bare integers fail normally where appropriate rather than rounding/overflowing internally.
6. Split alphabet-specific rule construction out of fjs/bnf/module.f.ts: Unicode helpers go to fjs/bnf/unicode/module.f.ts, binary byte-stream helpers go to fjs/bnf/byte/module.f.ts, and implicit string -> Unicode-code-point semantics disappear from the generic BNF/data path. Existing design TODOs that assume pre-split rule discriminants or helper ownership are explicitly blocked/rebased: the JSON grammar-owner TODO targets bnf/unicode; 207.md is explicitly blocked by the alphabet split and must be rebased/split after removing string as a generic rule kind; 667-bnf-repeat-flatten.md is blocked by the alphabet split, bigint symbol/range migration, and semantic-actions design because its previous bare-string Repeat and typeof rule === 'number' terminal dispatch are invalidated; rule-visitor.md is blocked by the alphabet split and bigint terminal/range migration and must define its visitor against the final post-migration Rule union; the recognizer/DFA TODO consumes bnf/byte helpers instead of defining a second binary-helper family; and proof-recognizer-and-fixtures.md is blocked by the split and must construct its shared text fixtures through bnf/unicode rather than core range. The older data-tosequence-reuse.md TODO is marked irrelevant because it is superseded by this split.
7. Move BNF symbols from 24-bit number to a fixed 256-bit bigint domain, reserving 2^256 - 1 as EOF. Keep the TerminalRange representation undecided until a separate investigation compares the simple fixed-width bigint baseline against bit-interleaved, variable-width, and structural representations. This is a serialized BNF-format decision, not only a runtime optimization: the fixed-width form remains the baseline and should win unless another option has a clear measured advantage, but the bigint migration stays blocked until that small comparison is made. Input adapters provide only physical ordinary symbols; each parser backend synthesizes exactly one logical EOF after physical input and tracks its one-time consumption only in internal parser state. Public positions/remainders stay in the physical input domain, so an EOF-consuming indexed match still reports input.length rather than input.length + 1. Parameterize the existing shared fjs/types/range_map algorithm by ordered boundary type/comparison/predecessor operations, preserve its current number API as a wrapper, and instantiate BNF dispatch with raw bigint cut points rather than the semantic Symbol domain, so an internal below-minimum cut point such as -1n is valid while parser inputs and terminal endpoints still satisfy the uint256 Symbol invariant.
8. Replace the registered fjs/bnf/token_symbol alphabet with deterministic token-name-derived symbols. The direct UTF-8 mapping uses tryUtf8, then a fallible tryToSentinel conversion that also handles the exact Bun bigint limit, and finally validates the actual candidate against the ordinary domain 0n <= symbol && symbol < eofSymbol. The 31-byte UTF-8 boundary is a derived property/proof, not a preflight size check. Alternative mappings such as cryptographic hashes must validate the same full uint256 ordinary-symbol invariant for every configured result, in addition to rejecting collisions over the concrete token alphabet. Existing token-parser designs are rebased before token_symbol is removed: new-parser.md is blocked by the bigint-symbol and token-symbol tasks, defines a finite ordinary DJS token-name alphabet, and maps ordinary token names through the same fallible Symbol mapping. Because the current DJS tokenizer emits a physical final eof token while BNF backends synthesize logical EOF, the parser adapter explicitly removes exactly one final physical eof before symbol mapping, preserves its real TokenMetadata separately as eofMetadata for end-of-input diagnostics, and never maps that physical EOF into the ordinary symbol stream. The new BNF parser targets the full current DJS module grammar, stays private inside the existing parser subsystem only while differential parity is proved, keeps parseFromTokens as the public API, and after success/failure parity switches that API to the BNF implementation while deleting the old hand-written state machine in the same task. At that cutover the DJS side of TODO 157 §1 is superseded; its independent serializer work is unaffected.

The dependency structure is explicit: lossless JSON starts at numeric tokenization, and the resulting exact NumberToken tree is the common substrate for extended JSON, standard FunctionalScript JSON, RTTI-aware validation, and any future compatibility work. Numeric edge cases block only the FunctionalScript codec policies that genuinely need those decisions; native equivalence does not block P3 work. Standard and extended parsers can materialize independently from the same exact structural tree, while already-materialized runtime values can still be converted with the reusable transformers. The alphabet/core split, extended JSON, and TerminalRange representation investigation precede bigint BNF symbols; token-symbol mapping then depends on the enlarged symbol domain, and the DJS parser cutover depends on both migrations plus differential parity before the old parser is removed.

The existing fjs/djs/todo/json-bigint-serialization.md is narrowed to DJS-specific integration and depends on the generic extended JSON layer rather than defining another parser/serializer.

Why

One JSON tokenizer and structural parser should preserve the information available in JSON text before a caller chooses a numeric policy. That requires tokenization itself to retain valid numeric lexemes independently of bounded derived numeric representations. Extended JSON, standard FunctionalScript JSON, and RTTI-aware parsing can then be different materializers/formatters over one exact structural core instead of parallel parsers. Exact native compatibility is deliberately postponed: after the core transforms exist, we can move json.* toward native behavior incrementally, including breaking changes when justified, without making compatibility a prerequisite for the architecture.

Likewise, the same BNF engine should work over different alphabets: bytes, Unicode code points, tokenizer output symbols, and future intermediate symbol streams. Core BNF therefore should not interpret JavaScript strings or byte containers as terminals. Unicode text and binary byte streams are adapters on top of the generic rule algebra.

A 256-bit finite symbol space keeps EOF/full-range/complement semantics simple, supports direct descriptive token-name mappings, and can also hold cryptographic-hash-based identifiers for arbitrarily large names. The representation of a range over that symbol space is treated as a separate format decision rather than inherited automatically from the old 24-bit packing.

Validation

Documentation/TODO-only change; no runtime code changed, so no test run was needed.