Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,13 @@ history.

## Unreleased

- `text/utf16`: `u16` now rejects non-integers, so a fractional word is
reported invalid (`0xFFFFFFFF`) instead of being misclassified by the
surrogate/BMP range checks — which only partition the integers in
`0x0000`–`0xFFFF`. That closes the only path into
`utf16ByteToCodePointOp`'s trailing fallback arm, which is removed along
with its `isHighSurrogate` recheck
[#1540](https://github.com/functionalscript/functionalscript/pull/1540)
- `media/json/parser`: `endArray`/`endObject` no longer branch on `state.top`
and `tokenToValue` drops its defensive default arm — the parser's state
machine already guarantees these invariants, so the dead branches are gone
Expand Down
15 changes: 12 additions & 3 deletions fjs/text/utf16/module.f.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,8 @@ const codePointToUtf16 = codePoint => {
export const fromCodePointList
= flatMap(codePointToUtf16)

const isInU16Range = contains(0x0000, 0xFFFF)

/**
* Validates whether a given 16-bit unsigned integer (U16) falls within the valid range for UTF-16 code units.
*
Expand All @@ -135,9 +137,14 @@ export const fromCodePointList
* const edgeCaseHigh = u16(0xFFFF) // true: Maximum valid value for UTF-16
* ```
*
* `U16` is just `number`, so this also rejects non-integers — `isBmpCodePoint`,
* `isHighSurrogate`, and `isLowSurrogate` only partition the *integers* in
* `0x0000`–`0xFFFF` with no gap; a fractional value would fall between two of
* those ranges and be misclassified downstream if it were let through here.
*
* @type {(i: U16) => boolean}
*/
const u16 = contains(0x0000, 0xFFFF)
const u16 = i => Number.isInteger(i) && isInU16Range(i)


/**
Expand Down Expand Up @@ -191,9 +198,11 @@ const utf16ByteToCodePointOp = (word, state) => {
const low = word - 0xdc00
return [[(high << 10) + low + 0x10000], null]
}
// `isLowSurrogate`, `isBmpCodePoint`, and `isHighSurrogate` partition the
// full `u16` range with no gap, and `isLowSurrogate` was already ruled out
// above, so a non-BMP `word` here is always a high surrogate.
if (isBmpCodePoint(word)) { return [[state | errorMask, word], null] }
if (isHighSurrogate(word)) { return [[state | errorMask], word] }
return [[state | errorMask, word | errorMask], null]
return [[state | errorMask], word]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve rejection of fractional surrogate-gap values

U16 is only an alias for number, and u16 checks only the inclusive numeric range, so values between the surrogate ranges are reachable. For example, decoding [0xd800, 0xdbff + 0.5, 0xdc00] now treats the fractional middle value as a pending high surrogate and emits 0x10fc00 as if it were a valid pair; the removed fallback previously emitted masked errors for both invalid units. Retain a fallback or reject non-integers before unconditionally treating this branch as a high surrogate.

AGENTS.md reference: AGENTS.md:L401-L403

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in fe050dd — I verified the gap is real (contains is a plain <=/>= check, so e.g. 56319.5 matched none of isHighSurrogate/isLowSurrogate/isBmpCodePoint). Tightened u16 with Number.isInteger so the three predicates' exhaustive-partition guarantee actually holds for every input the function can receive, and added proof cases for a fresh and a pending-state fractional word.


Generated by Claude Code

Comment on lines +201 to +205

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Delete the resolved UTF-16 todo

This change implements fjs/text/utf16/todo/unreachable-pending-state-fallthrough.md, but that issue file remains open in the commit, so the repository's issue tracker continues to advertise already-completed work. Delete the resolved todo as part of this change.

AGENTS.md reference: AGENTS.md:L1170-L1176

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deleted in fe050dd.


Generated by Claude Code

Comment on lines +201 to +205

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Add the required changelog entry

This commit changes authored FunctionalScript code but leaves CHANGELOG.md unchanged, so the release notes will omit the UTF-16 decoder change. Add a short entry at the top of Unreleased with the real PR link as required for code changes.

AGENTS.md reference: AGENTS.md:L1194-L1203

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already added in b7ae07e, before this review ran — see CHANGELOG.md's ## Unreleased section, text/utf16 entry linking #1540.


Generated by Claude Code

}


Expand Down
15 changes: 15 additions & 0 deletions fjs/text/utf16/proof.f.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,21 @@ export const proof = {
() => {
const result = stringify(toArray(toCodePointList([56320, 0])))
assertEq(result, '[-2147427328,0]')
},
// `U16` is just `number`, so a non-integer in [0x0000, 0xFFFF] is a
// possible (if malformed) input. It must be rejected as invalid, not
// misclassified by the surrogate/BMP range checks, which only
// partition the integers in that range.
() => {
const result = stringify(toArray(toCodePointList([56319.5])))
assertEq(result, '[4294967295]')
},
// A non-integer word doesn't disturb a pending high surrogate: it is
// reported invalid on its own, and the surrogate is still flagged
// unpaired at EOF.
() => {
const result = stringify(toArray(toCodePointList([55296, 56319.5])))
assertEq(result, '[4294967295,-2147428352]')
}
],
fromCodePointList: [
Expand Down
79 changes: 0 additions & 79 deletions fjs/text/utf16/todo/unreachable-pending-state-fallthrough.md

This file was deleted.

Loading