diff --git a/DESIGN.md b/DESIGN.md index 2ee621b5a..c224dbc3a 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -107,8 +107,78 @@ on top of the weaker design. belongs in `fjs/path`, not inline in a loader). First search for an appropriate existing module; create a new one only if no good fit exists. This is different from DRY extraction: it is always appropriate. +- **Follow the example** — one skeleton for every context; differences live in + the parts it calls, and improvements go into the skeleton so everyone gets + them. See below. - **Avoid side effects and mutability.** +### Follow the example + +When a capability already exists somewhere in the repository and is being +brought to a second context — another host, another backend, another runner — +**the existing one is the specification.** + +What is shared is the **skeleton**: the control flow, the order of operations, +the decisions and their names — the shape of the whole thing. Every context runs +that same skeleton. Where a context differs, it differs by supplying a different +**part** that the skeleton calls out to, at a place the skeleton names. It does +not differ by having a skeleton of its own. + +So there are exactly two ways to accommodate a context, and both are additive: + +- **Adjust that context's part.** A browser writes rows into a DOM where a + terminal writes lines to stdout; those are two implementations of one named + part, and the skeleton above them cannot tell which it has. +- **Improve the skeleton, for everyone.** If what the new context needs is + something the skeleton should have had, put it there. Every context gets it, + and that is a feature of the change rather than a side effect to apologize + for. + +There is no third way. A branch inside the skeleton that asks which host it is +running on is a fork wearing a shared name, and it is worse than two honest +implementations, because nothing about the shared name signals the difference. A +context that cannot be served by any existing part means the skeleton is missing +an extension point: add the point — one more named part that every context then +supplies — rather than a special case. + +The order of work follows from that: + +1. **Share the skeleton.** Take the existing implementation as the core, with + its behaviour unchanged. +2. **Adjust the parts** the new context genuinely requires, or extend the + skeleton so it can express what the new context needs. +3. **Document every difference that remains,** at the part where it is made. +4. **Open an issue for each problem the port revealed,** rather than fixing it + inside the port. +5. **Solve each issue in the skeleton or in every part at once,** so the + contexts stay in sync. + +The parts worth stating outright: + +**Differences are allowed; undocumented differences are not.** The goal is not +one identical behaviour — a browser has no stdout and a terminal has no DOM, and +pretending otherwise invents a host that does not exist. The goal is that every +difference lives in a named part, is deliberate, and is traceable to something +the host forced. "This context could do better here" is not such a reason: that +is an improvement, and an improvement belongs in the skeleton, where everyone +gets it. + +**Solve it for every context, or for none.** Once an issue from step 4 is picked +up, the fix lands everywhere in the same change. A fix in one context only is how +the contexts drift back apart, and it hides the finding from the place that has +had the defect longest — usually the older one. + +**The example may be simple for a reason.** What looks like a gap from inside +the new context is often a decision made in the old one. Copy it first; if it +turns out to be wrong, it is wrong in both places and worth an issue that says +so. + +**Keep the port separate from everything it inspires.** Land the sharing change +on its own, with behaviour unchanged. Anything new — a different scheduling +policy, a better measurement, an extra guard — is its own change afterwards. +Combined, they cannot be reviewed: an argument about the new idea becomes an +argument about the port. + ### Exception to DRY: performance measurement Time measurement must capture immediately after an operation completes to avoid diff --git a/fjs/emergent_testing/todo/hostile-proof-values.md b/fjs/emergent_testing/todo/hostile-proof-values.md new file mode 100644 index 000000000..3bc46b646 --- /dev/null +++ b/fjs/emergent_testing/todo/hostile-proof-values.md @@ -0,0 +1,113 @@ +## Hostile thrown values and cross-realm promises kill a run + +**Priority:** P3 +**Status:** open + +### Problem + +The browser runner (`../browser.mjs`) defends against two things `fjs t` does +not, and neither is reachable from ordinary FunctionalScript. That asymmetry is +the point of this file: when the two runners are unified +([share the browser and console proof runners](share-browser-console-runner.md)), +the shared core has to have *one* answer for each of them, decided rather than +inherited twice. `fjs t` is the reference, so the honest reading is that these +are gaps in `fjs t` which the browser happened to cover — and closing them in +the shared core is the way to keep that coverage instead of losing it to a port. + +**A value that resists being read is not attributed to the test that produced +it.** Two functions in the shared core read user-supplied values without a +guard: the `collectTests` traversal enumerates a returned proof tree, and +`errorDetails` reads `message`/`stack` and calls `String` on a thrown value. A +throwing accessor, a revoked `Proxy`, or a hostile `toString` panics through +either, and there is no `try`/`catch` in FunctionalScript for the core to catch +it with. `fjs t` ends with a stack trace and no summary; the browser runner +today loses one test and carries on. What is missing from the core is +*attribution*: naming the leaf whose value could not be read, and continuing +with the rest. Whichever runner ends up on top of it, a page left in `running` +or a process that exits with no summary is the outcome an automated controller +cannot act on. + +**A promise from another realm is not awaited.** `fjs t`'s `sandbox` asks `p +instanceof Promise`, which is false for a promise built in an iframe, a worker, +or a `node:vm` context. Such a value is walked as an ordinary proof tree +instead, so a *rejected* cross-realm promise is reported as a pass. The browser +runner carries `Symbol.species` machinery against this, which is a second answer +to the same question and is studied in +[imports, promises and realms](imports-promises-realms.md). The obvious repair — +brand-checking with `Object.prototype.toString` — is not one: the tag is +settable through `Symbol.toStringTag`, and an object carrying a `then` proof +would then be assimilated, breaking the rule that only actual promises are +asynchronous values. + +### Design: a `catch` operation + +Reading a user value belongs to the *operation*, not to the shared core, which +is what makes one fix serve every runner. Once the two runners share a core, +guarding the traversal once covers `fjs t` and the browser together — which is +an argument for doing this *with* the sharing change rather than before it. + +**`sandbox` cannot hold it, and the reason is not the one it looks like.** +Timing is not the obstacle: the sub-tree walk in `runModule` happens *after* the +runner has resolved the leaf's promise, so `sandbox(() => collectTests(path, +false, r))` would run a pure synchronous thunk over an already-settled value. +The obstacle is the **virtual runner**. Its `sandbox` is a deliberate +pass-through — `f => state => [state, ok(f())]`, with the fixture returning the +`SandboxResult` it wants reported — because `../../effects/node/virtual` is +`.f.mjs` and FunctionalScript has no `try`/`catch` to implement a real one with. +Routing the traversal through `sandbox` would hand that handler a thunk +answering `_TestAndPath[]`, which it would cast to `SandboxResult` and every +fixture in `../proof.f.mjs` would break. + +So add a second, honest operation beside it: + +```ts +export type Catch = readonly['catch', (f: () => T) => OpResult>] +``` + +"Run this pure thunk; a throw is the `error` branch." It carries no clock and no +fixture convention, so each runner implements it truthfully: + +- the real Node runner, and whatever browser interpreter the sharing change + produces: `tryCatch(f)`, one line each, from `types/result/module.mjs`. +- `effects/node/virtual/module.f.mjs`: `ok(ok(f()))` — a pure runner still + cannot catch, and a hostile fixture still panics there, which is the same + bargain `sandbox` already makes. Virtual proofs use benign fixtures. + +`walk` then reads a sub-tree through `catch` and, on the `error` branch, reports +one failed result at that path instead of panicking — which is what restores +`exportedTreeThrows` / `returnedTreeThrows`, and gives `fjs t` a behaviour it +never had. `errorDetails` gets the same treatment at its one call site. + +The work is roughly: the operation and its constructor beside `sandbox`, one +handler in each runner, the `CommandSet` entries, the `walk` change and its new +result shape, and the mock maps in the affected proofs. + +**The brand check** for cross-realm promises is not designed here. It belongs +with the two mechanisms it keeps being confused with — a module namespace +adopting a `then`, and a proof tree refusing to — which are studied together in +[imports, promises and realms](imports-promises-realms.md). + +### Tasks + +- [ ] Add the `catch` operation, its constructor, and a handler in each of the + Node, browser and virtual runners. +- [ ] Read sub-trees through it in `walk`, reporting an unreadable tree as one + failed result at its path rather than a panic. +- [ ] Prove an unreadable exported tree and an unreadable returned tree, for + `fjs t` as well as the browser — the browser has versions of these today + and `fjs t` has none. +- [ ] Read a thrown value through it at `errorDetails`' call site. + +### Constraints + +- Whatever is added must apply to `fjs t` and to the browser runner alike. A + defense in one runner only is the thing to avoid: it is how the two came to + mean different things in the first place. +- An object carrying a `then` proof property must stay an ordinary proof tree. + +### Related + +- [Imports, promises and realms](imports-promises-realms.md) — where the + cross-realm brand check is studied. +- [Browser testing](browser-testing.md) +- [Test-runner behavior](661-test-runner-behavior.md) diff --git a/fjs/emergent_testing/todo/imports-promises-realms.md b/fjs/emergent_testing/todo/imports-promises-realms.md new file mode 100644 index 000000000..0cfe8f249 --- /dev/null +++ b/fjs/emergent_testing/todo/imports-promises-realms.md @@ -0,0 +1,78 @@ +## Investigate imports, promises and realms + +**Priority:** P3 +**Status:** open — investigation, not yet actionable + +### Problem + +Three mechanisms meet in the runner, none of them is written down as a rule, and +the code where they meet reads as a pile of special cases rather than a design. +They are separate mechanisms that happen to interact, and the interaction is +what nobody has stated: + +**A module namespace object is a thenable.** `import()` resolves by *adopting* +what a module exports, so a module exporting a function named `then` corrupts +its own dynamic import. That is why exporting `then` from a proof module is +forbidden ([`spec/todo/3240-export.md`](../../../spec/todo/3240-export.md)) — +but the rule lives in a spec issue and a README paragraph, and nothing checks +it. The proof discovery in `../../dev/module.f.mjs` imports whatever it finds. + +**A proof tree is not a thenable, even when it has a `then`.** The runner's rule +is that only an actual `Promise` is an asynchronous value, so `{ then: f }` +returned from a proof is a sub-tree with a test called `then` in it. This is the +opposite reading of the same property name, one layer down, and both readings +are correct in their own layer. Nothing says so in one place. + +**`instanceof Promise` is realm-local.** A promise built in an iframe, a worker +or a `node:vm` context is not `instanceof Promise` here, so under `fjs t` it is +walked as a proof tree and a *rejected* one is reported as a pass. The browser +runner defends against this with `Symbol.species` shadowing and an intrinsic +`then` — about 150 lines (`../browser.mjs`, `../browser/species.proof.mjs`) that +read as a magic mess and are, today, the only place the exposure is covered. So +the two runners answer this question differently, and +[sharing them](share-browser-console-runner.md) forces a single answer: keep the +machinery, replace it with something statable, or accept `fjs t`'s exposure +knowingly. Deciding that by default, inside a port, is how the coverage gets +lost without anyone choosing to lose it. + +The three are usually discussed one at a time, which is why the interaction +keeps being rediscovered: the thing that makes a namespace dangerous (`then` is +adopted) is the thing the runner deliberately refuses to do (`then` is a name), +and the check that separates them (`instanceof`) is the one that does not +survive a realm boundary. + +### What to investigate + +This is a study, not a design. It is worth doing before +[browser-testing](browser-testing.md) puts proofs in iframes or workers, because +that is the point at which cross-realm promises stop being hypothetical. + +- **State the layering.** One document saying which layer adopts a `then` and + which layer refuses to, and why both are right. Until that exists, every fix + to one looks like a bug in the other. +- **Find a brand check that survives a realm and cannot be forged.** + `Object.prototype.toString` is forgeable through `Symbol.toStringTag`. + `Promise.resolve(p) === p` against the value's own constructor is a candidate. + Whatever is chosen must be one function every interpreter calls. +- **Decide whether the runner should see namespace objects at all.** If + discovery handed the runner a plain record of proofs rather than the module + namespace, the `then` export hazard would not reach it — and the `then`-export + ban could become a check rather than a convention. +- **Establish what the 150 lines actually buy**, from the proofs that cover them + (`../browser/species.proof.mjs`), so that whatever replaces them is measured + against the same cases rather than against a memory — and so that removing + them, if that is the answer, is a decision with a list attached. + +### Constraints + +- An object carrying a `then` proof property must stay an ordinary proof tree. +- Whatever is added must apply to every runner. A defence in one host only is + the state this is trying to leave. + +### Related + +- [Hostile proof values](hostile-proof-values.md) — the cross-realm promise + exposure, and the traversal guard it shares a cause with. +- [Browser testing](browser-testing.md) — iframes and workers. +- [`spec/todo/3240-export.md`](../../../spec/todo/3240-export.md) — the `then` + export ban. diff --git a/fjs/emergent_testing/todo/report-before-running.md b/fjs/emergent_testing/todo/report-before-running.md new file mode 100644 index 000000000..37d8fff98 --- /dev/null +++ b/fjs/emergent_testing/todo/report-before-running.md @@ -0,0 +1,79 @@ +## Report a test's name before running it, not only after + +**Priority:** P2 +**Status:** open + +### Problem + +Every runner reports a test only once it has finished. `fjs t` writes +`import("./a.proof.f.mjs").proof.x(): ok, 0.3 ms` after the fact, and the +browser page appends `PASS import("a").proof.x() (0.3 ms)` the same way. A test +that is *running* is invisible. + +Three things follow from that, and the third is the one that matters: + +- **A slow test looks like a hung runner.** Nothing distinguishes "this proof + has been going for ten seconds" from "the runner stopped", so the only way to + find the slow one is to wait for it to finish and read the duration. +- **Progress is a count, not a place.** The browser page says "1247 tests + completed…" while a reader wants to know *which* one it is on. +- **A crash loses the one fact worth having.** When a proof takes the process + down — a panic through the shared traversal, an out-of-memory, a stack + overflow, a runner bug — the last line printed is the last test that + *succeeded*, and the one that actually broke is never named. That is exactly + the case where a name is worth more than a result, and it is the case where + the current design has none. + +No reporter has an event for it: `result` is called with a `SandboxResult`, so it +cannot be called before there is one. + +### Preliminary design + +Add a `start` (or `begin`) event to the reporter, called with the file and path +before the leaf is sandboxed, and let each host decide what to do with it: + +- **`fjs t`** prints the name, then completes the line with `ok`/`error` and the + duration when the result lands — the standard runner shape, in the format it + already prints. Interleaving is the thing to get right: leaves + run concurrently, so a half-written line cannot be left open across another + test's output. Either the name and its outcome are one deferred line with the + name shown live elsewhere, or output is a two-column log that names the start + and closes it by identifier. +- **The browser page** renders a row in a pending state and settles it in place, + which is the same list it renders now with one more state per row. +- **A result type** may not need to change at all: a start is an event, not a + result. Whether the reporter grows a sibling operation or its existing one + gains a status is part of the design. + +The reporter change is small; the interleaving question is the real one, and +it is the same question in both hosts, which is an argument for settling it in +the shared core rather than twice. + +### Constraints + +- A start event must not cost a `sandbox` call or a clock read of its own: the + duration reported is still the sandboxed one. +- Concurrency stays. Naming a test before running it must not serialize the + suite to keep the output tidy. +- Whatever is emitted has to be as useful to an automated consumer as to a + reader — a start with no matching result is precisely the signal a crashed + run leaves behind, and a controller should be able to read it. +- The start event lands in both runners in the same change. Their output differs + — a terminal line and a DOM row — but a runner that names a running test and + one that does not are two different tools. + +### Tasks + +- [ ] Add the start event to the reporter and call it before the + leaf is sandboxed. +- [ ] Decide the terminal format for concurrent output, and prove it. +- [ ] Render a pending row in the browser page and settle it in place. +- [ ] Prove that a run killed mid-test leaves the running test's name behind. + +### Related + +- [Share the browser and console proof runners](share-browser-console-runner.md) + — reporting is one of the things each host still does its own way, and this + is the same question twice until they share a reporter. +- [Hostile proof values](hostile-proof-values.md) — the crash case this would + make diagnosable, where today the run ends with no summary and no name. diff --git a/fjs/emergent_testing/todo/share-browser-console-runner.md b/fjs/emergent_testing/todo/share-browser-console-runner.md index 771806229..5e3cc0e56 100644 --- a/fjs/emergent_testing/todo/share-browser-console-runner.md +++ b/fjs/emergent_testing/todo/share-browser-console-runner.md @@ -21,6 +21,84 @@ The current browser file also mixes three layers: That makes the reusable semantics harder to see and leaves the impure browser entry much larger than it needs to be. +### How to do this — read before designing + +An attempt at this issue was written, reviewed, approved and then reverted. It +worked: one shared `runModuleMap`, a `Reporter` per host, an `effects/common` +layer, a browser interpreter, 100% coverage, green CI, a real Chromium run of +3435 proofs. It was reverted anyway, because *how* it got there is a cost this +repository does not want to pay again, and the record of why is worth more than +the code was. **The order of work is the deliverable here, not just the final +shape.** See [DESIGN.md §4, "Follow the example"](../../../DESIGN.md). + +**One skeleton, with named parts.** The thing to share is the *runner itself*: +the order in which modules are linked, leaves discovered, bodies executed, +throws inverted, results counted and the run concluded. Both hosts run that same +skeleton. Everything host-specific is a **part** the skeleton calls at a place it +names — where the leaf body is executed, where a result is reported, where a +module is linked — and a part is where a browser is allowed to be a browser. + +That gives exactly two ways to accommodate a host, both additive: change *that +host's part*, or *improve the skeleton so every host benefits*. There is no +third. A branch inside the skeleton asking which host it is running on is a fork +wearing a shared name. A host need that no existing part can express means the +skeleton is missing an extension point — add the point, which every host then +supplies, rather than a special case. + +Differences between the parts are fine and expected: a DOM row and a terminal +line are two implementations of the same named part, and the skeleton above them +cannot tell which it has. *Undocumented* differences are not. The attempt shared +the modules and then let the browser keep its own test-name format, its own +scheduling policy and its own clock — none of which its host forced, and none of +which belonged in a part. That is the failure mode: it *looks* like success — +one module, one name — while two behaviours hide behind it, and two +implementations behind two names would have been more honest, because nothing +about the shared name signals the difference. + +**`fjs t` is sequential, and that is a decision to copy, not a gap to fill.** +The attempt gave the browser a batch size — proofs launched in groups with a +yield between groups. Nobody had asked for it, no measurement motivated the +constant, and it was premature optimization in the strict sense: it made the +runner different from the example in order to solve a problem no one had +reported. Everything that followed was created by that choice. In order: the +batching had no paint boundary where it claimed one; the fix serialized the +groups and deadlocked a graph `fjs t` completes; the proof written for *that* +fix was flaky under load; its rewrite passed for the wrong reason and had to be +made first-opener-wins; the yield needed `MessageChannel` rather than +`setTimeout` only because `setTimeout` clamps to 4 ms once nested; and the +`MessageChannel` proof then failed under bun, which drains port messages before +running a due timer. Six rounds of review, every one of them downstream of a +constant that was finally deleted. The end state — no batch size at all — is +the state that copying `fjs t` would have produced on day one. + +**A problem the browser reveals is not a browser problem.** Two came up, and +both are properly issues rather than fixes inside a port: + +- Timing. `performance.now()` is coarsened and jittered in browsers, so a + per-proof duration there is largely the clamp. But `sandbox` is the shared + operation, so this is one decision for both hosts, not a browser-local + workaround. See [Browser timer precision](timer-precision.md). +- Hostile values and cross-realm promises. The browser file today carries + defenses `fjs t` has never had. Sharing the core means deciding what the rule + *is*, once — not quietly keeping two. See + [Hostile thrown values and cross-realm promises](hostile-proof-values.md) and + [Imports, promises and realms](imports-promises-realms.md). + +The rule that follows: **land the shared skeleton with behaviour unchanged, then +take each new problem as its own change — in the skeleton where it belongs +there, so both runners get it, or in every part at once.** An improvement the +browser could have is an issue, not something to introduce inside a port. A +behaviour the port cannot preserve is a finding to record before it merges, not +a silent divergence to explain in review. + +**Keep the change reviewable.** The attempt was 2646 insertions and 1408 +deletions across 35 files in one PR — a move, a rewrite, a new effects layer, a +new host interpreter and a scheduling invention at once, which is why the +scheduling argument could not be separated from the sharing argument. Sequence +it: the shared semantics first, with `fjs t` unchanged in behaviour and the +browser file only calling into it; the layout moves after; anything genuinely +new last, on its own. + ### Preliminary design Share semantics, not host mechanics. The console runner should keep using the @@ -57,7 +135,9 @@ script. If preparation needs a Node capability that the FunctionalScript program cannot currently express, add the smallest operation to `fjs/effects/node/` and its real and virtual interpreters instead of bypassing Effects. Existing `readdir`, `readFile`, and `writeFile` operations should be -reused where sufficient. +reused where sufficient. This is the build rather than the runner, and it is a +reasonable second change rather than part of the first one — but it is part of +this issue, so it does not get dropped on the way. Move `emergent_testing/browser.mjs` to `emergent_testing/browser/module.mjs`. It should become a thin impure shell: @@ -74,7 +154,13 @@ Extract or reuse these host-independent concepts first: - expected-throw semantics; - normalized per-test results and total/result reducers; - report status and infrastructure-error classification; -- semantic progress events, independent of terminal text or DOM elements. +- semantic progress events, independent of terminal text or DOM elements; +- **the test name.** `fjs t` prints + `import("./a.proof.f.mjs").proof.x(): ok, 0.3 ms`, and the browser page must + produce the same identifier for the same leaf. A shared core that leaves each + host to format its own name has not finished sharing: a name is what makes two + reports comparable, and a divergence there is the visible proof that the + semantics underneath were never actually unified. Keep host capabilities at the leaves. Candidate browser effects are module import, monotonic time, event-loop yield, and report publication. DOM node @@ -83,6 +169,16 @@ an effect adds an operation for every DOM detail without improving the shared API. Add `fjs/effects/browser/` only after the required operation set is clear; do not create a mirror of `effects/node` merely for directory symmetry. +A shared `all` that starts every child before awaiting any is worth stating as a +contract rather than leaving to each interpreter: a child may wait on something +a later sibling produces, so an interpreter that awaits one child before +starting the next hangs a graph the other host completes. Beyond that, **the +browser gets no scheduling policy of its own until someone reports a problem +with the one `fjs t` has.** If a page turns out to need a task boundary to +paint, that is a separate, measured change with its own issue — and the measure +is a boundary per unit of work, never a tuned count of proofs, because proofs +differ in cost by orders of magnitude. + An executor boundary will still be necessary because the console runner uses the Effects sandbox while a browser catches synchronous throws and awaits native promises. That boundary should answer one normalized leaf result. Tree @@ -94,6 +190,17 @@ are shared. - Preserve the recursive proof semantics and totals of `fjs t` exactly, including objects with a proof property named `then`; only actual promises are asynchronous values. +- Both runners must produce the same test name for the same leaf. This one is + not a host difference: nothing about a browser prevents it, and a divergence + here is the visible sign that the semantics underneath were never unified. +- The skeleton never asks which host it is running on. Anything host-specific is + a part it calls; anything it cannot express through a part is a missing + extension point, not a special case. +- Every remaining difference between the two runners lives in a part, is + documented there, and is traceable to something the host forced. Host APIs and + wrappers may differ freely; behaviour may differ only for a written reason. +- A fix for a problem either runner has lands in the skeleton, or in every part + at once — in the same change. - Browser modules must not import Node built-ins, the Node effect interpreter, `node:test`, or Playwright. - Website build-time filesystem access must be expressed by the FunctionalScript @@ -103,6 +210,10 @@ are shared. bundling or transpilation. - Pure `.f.mjs` additions require co-located proofs with complete line, function, and branch coverage. +- A proof must assert the property, not an engine's incidental scheduling. The + suite runs under node, deno and bun, and they do not agree on the ordering of + timers against other task sources — asserting one of those orderings makes a + correct implementation fail somewhere. - Keep the serializable browser report, documented promise, and completion event compatible unless a simpler shared report API deliberately replaces all callers in the same change. @@ -113,8 +224,13 @@ are shared. - [ ] Inventory duplicated semantics in `emergent_testing/module.f.mjs` and `emergent_testing/browser.mjs`, and define the smallest shared API. +- [ ] Name the skeleton's parts explicitly — execute a leaf, report a result, + link a module — and check that nothing host-specific is left outside one + of them. - [ ] Make the existing `collectTests`/path behavior the single source of truth for console and browser execution. +- [ ] Share the test-name format, and prove both runners name the same leaf + identically. - [ ] Define normalized leaf, progress, infrastructure-error, totals, and report values without terminal or DOM fields. - [ ] Decide whether browser import/time/yield/publication justify @@ -135,6 +251,10 @@ are shared. to the new module paths. - [ ] Prove both runners produce equivalent paths, throw outcomes, recursive test counts, and normalized failures from the same fixtures. +- [ ] Record every behaviour the browser file has today and the shared core will + not keep, as an issue, before the sharing change merges. +- [ ] Close each of those issues for both runners at once, so the two stay in + sync rather than drifting from the day the core is shared. ### Related @@ -144,3 +264,12 @@ are shared. that must remain intentional after sharing the core. - [Test tree walker](65z-tf-test-tree-walker.md) — earlier work around recursive proof-tree traversal. +- [Hostile thrown values and cross-realm promises](hostile-proof-values.md) — + a behaviour the browser has and `fjs t` does not; decide it, do not inherit + two answers. +- [Imports, promises and realms](imports-promises-realms.md) — the same, for the + loading and promise-detection machinery. +- [Browser timer precision](timer-precision.md) — `sandbox` is shared, so its + measurement is one decision for both hosts. +- [Report a test's name before running it](report-before-running.md) — + reporting is the next thing worth sharing after the semantics. diff --git a/fjs/emergent_testing/todo/timer-precision.md b/fjs/emergent_testing/todo/timer-precision.md new file mode 100644 index 000000000..edb330b27 --- /dev/null +++ b/fjs/emergent_testing/todo/timer-precision.md @@ -0,0 +1,104 @@ +## Browser timer precision makes per-proof durations mostly noise + +**Priority:** P2 +**Status:** open + +### Problem + +`sandbox` measures a proof by reading the clock either side of the body: + +```js +const before = performance.now() +// ... +return { result, duration: after - before } +``` + +That is right for `fjs t`, where `performance.now()` resolves to well under a +microsecond. It is not right in a browser, where the same call is deliberately +degraded as a Spectre and fingerprinting mitigation: + +- Chromium coarsens `performance.now()` to **100 µs** on an ordinary page, and + to 5 µs only when the page is cross-origin isolated (`COOP`/`COEP`). +- Firefox rounds to **1 ms** by default (`privacy.reduceTimerPrecision`) and + additionally *jitters* the value, so successive reads are not merely coarse + but non-deterministic. +- WebKit coarsens as well, and the exact figure has moved between releases. + +The numbers our own suite produces put almost every proof under those clamps: +a typical leaf in the CLI report is 0.03–0.2 ms. On an ordinary Chromium page +that is one clock tick or zero, and on Firefox it is zero or one whole +millisecond of jitter. So the browser page's `(0.3 ms)` column is not a +measurement of anything — it is the clamp, rendered per row. Worse, a *total* +built by summing thousands of such rows accumulates the rounding rather than +cancelling it, so the sum can be off by a large multiple in either direction +depending on which way each read rounded. + +Note this is not the same concern as monotonicity. `performance.now()` cannot +go backwards, which is why it is the right clock for a duration; a wall clock +would be worse. A monotonic clock can still be a coarse one, and this is about +the resolution. + +### Preliminary design + +Nothing here is decided; the point of the todo is to establish what is true +before changing the measurement. + +- **Measure the clamp rather than assume it.** A proof that reads the clock in + a tight loop and reports the smallest non-zero difference tells us the real + resolution in whatever browser is running, which is a fact the report could + carry alongside the durations. It is also the honest precondition for every + option below. +- **Report a resolution, not just a duration.** If the host clock ticks at + 100 µs, a row saying `0.1 ms` is claiming precision it does not have. The + report is serializable and consumed by controllers, so a `resolution` field + would let a consumer decide what is significant instead of guessing. +- **Accumulate over a group.** The idea raised when this was filed: time a + batch of leaves with one pair of reads and divide, so the clamp is amortized + across many proofs instead of applied to each. This is speculation — it + trades a per-test number for an average, it cannot attribute a slow proof, + and it interacts with concurrency, since `all` interleaves launches and a + group's wall time would then include siblings' work. Worth prototyping, + not worth assuming. +- **Cross-origin isolation.** Serving the eventual application root with + `COOP: same-origin` and `COEP: require-corp` buys Chromium's 5 µs clock and + is a header change in the shared controller, not a design change. It does + nothing for Firefox's jitter, and it constrains what the page may embed. +- **Consider not reporting a per-proof duration in the browser at all** if + none of the above yields a number worth printing. A column that is always + the clamp is worse than no column. + +### Constraints + +- `sandbox` is the operation that executes a proof body, and both runners must + agree on it exactly or a suite means different things in different hosts. Any + change to how it measures is a change for both, not a browser-local tweak — + and it is very likely `fjs t` has a milder version of the same problem, since + a coarse clock is only easier to notice in a browser. +- The clock must stay monotonic. Whatever replaces or supplements + `performance.now()` cannot reintroduce wall-clock time. +- A duration must not cost a second `sandbox` call or an extra scheduling + boundary: the reads are adjacent today precisely so a scheduler cannot + interleave between them. +- Whatever the browser reports has to stay serializable and comparable to what + `fjs t` reports, or the two reports cannot be diffed. + +### Tasks + +- [ ] Measure the actual `performance.now()` resolution in Chromium, Firefox + and WebKit from inside the runner, and record the figures here. +- [ ] Decide whether the report carries the resolution, and whether a row + below it renders a duration at all. +- [ ] Prototype accumulated timing over a group of leaves and check what it + costs in attribution and what concurrency does to it. +- [ ] Check whether cross-origin isolation is worth the headers in the shared + controller. + +### Related + +- [Run FunctionalScript proofs inside real browsers](browser-testing.md) — the + report contract these durations belong to. +- [Report a test's name before running it](report-before-running.md) — the + other thing wrong with what a row shows. +- [Share the browser and console proof runners](share-browser-console-runner.md) + — `sandbox` is the operation that executes a proof body in both hosts, so its + measurement is one decision, not two.