-
-
Notifications
You must be signed in to change notification settings - Fork 6
emergent_testing: keep what the shared-runner attempt taught, revert the code #1737
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
01f29ad
62ba9d1
b457587
0aac225
dc10dcf
57e295b
68fc983
505766d
569636b
4923ddb
cbc4454
0773613
401f191
5a74d4d
09bac1f
6f5dd59
565f4ce
fe112e8
b9b273c
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| ## Hostile thrown values and cross-realm promises kill a run | ||
|
|
||
| **Priority:** P3 | ||
| **Status:** open | ||
|
|
||
| ### Problem | ||
|
|
||
| The browser runner (`../browser.mjs`) defends against two things `fjs t` does | ||
| not, and neither is reachable from ordinary FunctionalScript. That asymmetry is | ||
| the point of this file: when the two runners are unified | ||
| ([share the browser and console proof runners](share-browser-console-runner.md)), | ||
| the shared core has to have *one* answer for each of them, decided rather than | ||
| inherited twice. `fjs t` is the reference, so the honest reading is that these | ||
| are gaps in `fjs t` which the browser happened to cover — and closing them in | ||
| the shared core is the way to keep that coverage instead of losing it to a port. | ||
|
|
||
| **A value that resists being read is not attributed to the test that produced | ||
| it.** Two functions in the shared core read user-supplied values without a | ||
| guard: the `collectTests` traversal enumerates a returned proof tree, and | ||
| `errorDetails` reads `message`/`stack` and calls `String` on a thrown value. A | ||
| throwing accessor, a revoked `Proxy`, or a hostile `toString` panics through | ||
| either, and there is no `try`/`catch` in FunctionalScript for the core to catch | ||
| it with. `fjs t` ends with a stack trace and no summary; the browser runner | ||
| today loses one test and carries on. What is missing from the core is | ||
| *attribution*: naming the leaf whose value could not be read, and continuing | ||
| with the rest. Whichever runner ends up on top of it, a page left in `running` | ||
| or a process that exits with no summary is the outcome an automated controller | ||
| cannot act on. | ||
|
|
||
| **A promise from another realm is not awaited.** `fjs t`'s `sandbox` asks `p | ||
| instanceof Promise`, which is false for a promise built in an iframe, a worker, | ||
| or a `node:vm` context. Such a value is walked as an ordinary proof tree | ||
| instead, so a *rejected* cross-realm promise is reported as a pass. The browser | ||
| runner carries `Symbol.species` machinery against this, which is a second answer | ||
| to the same question and is studied in | ||
| [imports, promises and realms](imports-promises-realms.md). The obvious repair — | ||
| brand-checking with `Object.prototype.toString` — is not one: the tag is | ||
| settable through `Symbol.toStringTag`, and an object carrying a `then` proof | ||
| would then be assimilated, breaking the rule that only actual promises are | ||
| asynchronous values. | ||
|
|
||
| ### Design: a `catch` operation | ||
|
|
||
| Reading a user value belongs to the *operation*, not to the shared core, which | ||
| is what makes one fix serve every runner. Once the two runners share a core, | ||
| guarding the traversal once covers `fjs t` and the browser together — which is | ||
| an argument for doing this *with* the sharing change rather than before it. | ||
|
|
||
| **`sandbox` cannot hold it, and the reason is not the one it looks like.** | ||
| Timing is not the obstacle: the sub-tree walk in `runModule` happens *after* the | ||
| runner has resolved the leaf's promise, so `sandbox(() => collectTests(path, | ||
| false, r))` would run a pure synchronous thunk over an already-settled value. | ||
| The obstacle is the **virtual runner**. Its `sandbox` is a deliberate | ||
| pass-through — `f => state => [state, ok(f())]`, with the fixture returning the | ||
| `SandboxResult` it wants reported — because `../../effects/node/virtual` is | ||
| `.f.mjs` and FunctionalScript has no `try`/`catch` to implement a real one with. | ||
| Routing the traversal through `sandbox` would hand that handler a thunk | ||
| answering `_TestAndPath[]`, which it would cast to `SandboxResult` and every | ||
| fixture in `../proof.f.mjs` would break. | ||
|
|
||
| So add a second, honest operation beside it: | ||
|
|
||
| ```ts | ||
| export type Catch = readonly['catch', <T>(f: () => T) => OpResult<Result<T, unknown>>] | ||
| ``` | ||
|
|
||
| "Run this pure thunk; a throw is the `error` branch." It carries no clock and no | ||
| fixture convention, so each runner implements it truthfully: | ||
|
|
||
| - the real Node runner, and whatever browser interpreter the sharing change | ||
| produces: `tryCatch(f)`, one line each, from `types/result/module.mjs`. | ||
| - `effects/node/virtual/module.f.mjs`: `ok(ok(f()))` — a pure runner still | ||
| cannot catch, and a hostile fixture still panics there, which is the same | ||
| bargain `sandbox` already makes. Virtual proofs use benign fixtures. | ||
|
|
||
| `walk` then reads a sub-tree through `catch` and, on the `error` branch, reports | ||
| one failed result at that path instead of panicking — which is what restores | ||
| `exportedTreeThrows` / `returnedTreeThrows`, and gives `fjs t` a behaviour it | ||
| never had. `errorDetails` gets the same treatment at its one call site. | ||
|
Comment on lines
+76
to
+79
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When AGENTS.md reference: AGENTS.md:L47-L51 Useful? React with 👍 / 👎. |
||
|
|
||
| The work is roughly: the operation and its constructor beside `sandbox`, one | ||
| handler in each runner, the `CommandSet` entries, the `walk` change and its new | ||
| result shape, and the mock maps in the affected proofs. | ||
|
|
||
| **The brand check** for cross-realm promises is not designed here. It belongs | ||
| with the two mechanisms it keeps being confused with — a module namespace | ||
| adopting a `then`, and a proof tree refusing to — which are studied together in | ||
| [imports, promises and realms](imports-promises-realms.md). | ||
|
|
||
| ### Tasks | ||
|
|
||
| - [ ] Add the `catch` operation, its constructor, and a handler in each of the | ||
| Node, browser and virtual runners. | ||
| - [ ] Read sub-trees through it in `walk`, reporting an unreadable tree as one | ||
| failed result at its path rather than a panic. | ||
| - [ ] Prove an unreadable exported tree and an unreadable returned tree, for | ||
| `fjs t` as well as the browser — the browser has versions of these today | ||
| and `fjs t` has none. | ||
| - [ ] Read a thrown value through it at `errorDetails`' call site. | ||
|
|
||
| ### Constraints | ||
|
|
||
| - Whatever is added must apply to `fjs t` and to the browser runner alike. A | ||
| defense in one runner only is the thing to avoid: it is how the two came to | ||
| mean different things in the first place. | ||
| - An object carrying a `then` proof property must stay an ordinary proof tree. | ||
|
|
||
| ### Related | ||
|
|
||
| - [Imports, promises and realms](imports-promises-realms.md) — where the | ||
| cross-realm brand check is studied. | ||
| - [Browser testing](browser-testing.md) | ||
| - [Test-runner behavior](661-test-runner-behavior.md) | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| ## Investigate imports, promises and realms | ||
|
|
||
| **Priority:** P3 | ||
| **Status:** open — investigation, not yet actionable | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The repository's todo format restricts AGENTS.md reference: AGENTS.md:L55-L56 Useful? React with 👍 / 👎. |
||
|
|
||
| ### Problem | ||
|
|
||
| Three mechanisms meet in the runner, none of them is written down as a rule, and | ||
| the code where they meet reads as a pile of special cases rather than a design. | ||
| They are separate mechanisms that happen to interact, and the interaction is | ||
| what nobody has stated: | ||
|
|
||
| **A module namespace object is a thenable.** `import()` resolves by *adopting* | ||
| what a module exports, so a module exporting a function named `then` corrupts | ||
| its own dynamic import. That is why exporting `then` from a proof module is | ||
| forbidden ([`spec/todo/3240-export.md`](../../../spec/todo/3240-export.md)) — | ||
| but the rule lives in a spec issue and a README paragraph, and nothing checks | ||
| it. The proof discovery in `../../dev/module.f.mjs` imports whatever it finds. | ||
|
|
||
| **A proof tree is not a thenable, even when it has a `then`.** The runner's rule | ||
| is that only an actual `Promise` is an asynchronous value, so `{ then: f }` | ||
| returned from a proof is a sub-tree with a test called `then` in it. This is the | ||
| opposite reading of the same property name, one layer down, and both readings | ||
| are correct in their own layer. Nothing says so in one place. | ||
|
|
||
| **`instanceof Promise` is realm-local.** A promise built in an iframe, a worker | ||
| or a `node:vm` context is not `instanceof Promise` here, so under `fjs t` it is | ||
| walked as a proof tree and a *rejected* one is reported as a pass. The browser | ||
| runner defends against this with `Symbol.species` shadowing and an intrinsic | ||
| `then` — about 150 lines (`../browser.mjs`, `../browser/species.proof.mjs`) that | ||
| read as a magic mess and are, today, the only place the exposure is covered. So | ||
| the two runners answer this question differently, and | ||
| [sharing them](share-browser-console-runner.md) forces a single answer: keep the | ||
| machinery, replace it with something statable, or accept `fjs t`'s exposure | ||
| knowingly. Deciding that by default, inside a port, is how the coverage gets | ||
| lost without anyone choosing to lose it. | ||
|
|
||
| The three are usually discussed one at a time, which is why the interaction | ||
| keeps being rediscovered: the thing that makes a namespace dangerous (`then` is | ||
| adopted) is the thing the runner deliberately refuses to do (`then` is a name), | ||
| and the check that separates them (`instanceof`) is the one that does not | ||
| survive a realm boundary. | ||
|
|
||
| ### What to investigate | ||
|
|
||
| This is a study, not a design. It is worth doing before | ||
| [browser-testing](browser-testing.md) puts proofs in iframes or workers, because | ||
| that is the point at which cross-realm promises stop being hypothetical. | ||
|
|
||
| - **State the layering.** One document saying which layer adopts a `then` and | ||
| which layer refuses to, and why both are right. Until that exists, every fix | ||
| to one looks like a bug in the other. | ||
| - **Find a brand check that survives a realm and cannot be forged.** | ||
| `Object.prototype.toString` is forgeable through `Symbol.toStringTag`. | ||
| `Promise.resolve(p) === p` against the value's own constructor is a candidate. | ||
| Whatever is chosen must be one function every interpreter calls. | ||
| - **Decide whether the runner should see namespace objects at all.** If | ||
| discovery handed the runner a plain record of proofs rather than the module | ||
| namespace, the `then` export hazard would not reach it — and the `then`-export | ||
| ban could become a check rather than a convention. | ||
| - **Establish what the 150 lines actually buy**, from the proofs that cover them | ||
| (`../browser/species.proof.mjs`), so that whatever replaces them is measured | ||
| against the same cases rather than against a memory — and so that removing | ||
| them, if that is the answer, is a decision with a list attached. | ||
|
|
||
| ### Constraints | ||
|
|
||
| - An object carrying a `then` proof property must stay an ordinary proof tree. | ||
| - Whatever is added must apply to every runner. A defence in one host only is | ||
| the state this is trying to leave. | ||
|
|
||
| ### Related | ||
|
|
||
| - [Hostile proof values](hostile-proof-values.md) — the cross-realm promise | ||
| exposure, and the traversal guard it shares a cause with. | ||
| - [Browser testing](browser-testing.md) — iframes and workers. | ||
| - [`spec/todo/3240-export.md`](../../../spec/todo/3240-export.md) — the `then` | ||
| export ban. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,79 @@ | ||
| ## Report a test's name before running it, not only after | ||
|
|
||
| **Priority:** P2 | ||
| **Status:** open | ||
|
|
||
| ### Problem | ||
|
|
||
| Every runner reports a test only once it has finished. `fjs t` writes | ||
| `import("./a.proof.f.mjs").proof.x(): ok, 0.3 ms` after the fact, and the | ||
| browser page appends `PASS import("a").proof.x() (0.3 ms)` the same way. A test | ||
| that is *running* is invisible. | ||
|
|
||
| Three things follow from that, and the third is the one that matters: | ||
|
|
||
| - **A slow test looks like a hung runner.** Nothing distinguishes "this proof | ||
| has been going for ten seconds" from "the runner stopped", so the only way to | ||
| find the slow one is to wait for it to finish and read the duration. | ||
| - **Progress is a count, not a place.** The browser page says "1247 tests | ||
| completed…" while a reader wants to know *which* one it is on. | ||
| - **A crash loses the one fact worth having.** When a proof takes the process | ||
| down — a panic through the shared traversal, an out-of-memory, a stack | ||
| overflow, a runner bug — the last line printed is the last test that | ||
| *succeeded*, and the one that actually broke is never named. That is exactly | ||
| the case where a name is worth more than a result, and it is the case where | ||
| the current design has none. | ||
|
|
||
| No reporter has an event for it: `result` is called with a `SandboxResult`, so it | ||
| cannot be called before there is one. | ||
|
|
||
| ### Preliminary design | ||
|
|
||
| Add a `start` (or `begin`) event to the reporter, called with the file and path | ||
| before the leaf is sandboxed, and let each host decide what to do with it: | ||
|
|
||
| - **`fjs t`** prints the name, then completes the line with `ok`/`error` and the | ||
| duration when the result lands — the standard runner shape, in the format it | ||
| already prints. Interleaving is the thing to get right: leaves | ||
| run concurrently, so a half-written line cannot be left open across another | ||
| test's output. Either the name and its outcome are one deferred line with the | ||
| name shown live elsewhere, or output is a two-column log that names the start | ||
| and closes it by identifier. | ||
| - **The browser page** renders a row in a pending state and settles it in place, | ||
| which is the same list it renders now with one more state per row. | ||
| - **A result type** may not need to change at all: a start is an event, not a | ||
| result. Whether the reporter grows a sibling operation or its existing one | ||
| gains a status is part of the design. | ||
|
|
||
| The reporter change is small; the interleaving question is the real one, and | ||
| it is the same question in both hosts, which is an argument for settling it in | ||
| the shared core rather than twice. | ||
|
|
||
| ### Constraints | ||
|
|
||
| - A start event must not cost a `sandbox` call or a clock read of its own: the | ||
| duration reported is still the sandboxed one. | ||
| - Concurrency stays. Naming a test before running it must not serialize the | ||
| suite to keep the output tidy. | ||
| - Whatever is emitted has to be as useful to an automated consumer as to a | ||
| reader — a start with no matching result is precisely the signal a crashed | ||
| run leaves behind, and a controller should be able to read it. | ||
| - The start event lands in both runners in the same change. Their output differs | ||
| — a terminal line and a DOM row — but a runner that names a running test and | ||
| one that does not are two different tools. | ||
|
|
||
| ### Tasks | ||
|
|
||
| - [ ] Add the start event to the reporter and call it before the | ||
| leaf is sandboxed. | ||
| - [ ] Decide the terminal format for concurrent output, and prove it. | ||
| - [ ] Render a pending row in the browser page and settle it in place. | ||
| - [ ] Prove that a run killed mid-test leaves the running test's name behind. | ||
|
|
||
| ### Related | ||
|
|
||
| - [Share the browser and console proof runners](share-browser-console-runner.md) | ||
| — reporting is one of the things each host still does its own way, and this | ||
| is the same question twice until they share a reporter. | ||
| - [Hostile proof values](hostile-proof-values.md) — the crash case this would | ||
| make diagnosable, where today the run ends with no summary and no name. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For hostile values thrown under
fjs t, this problem statement identifieserrorDetailsas an unguarded shared-core reader, but at this commit that helper is browser-only and already guards both property access andStringconversion (browser.mjs:24-57). The actual CLI coercions are the unguardedString(v)and template interpolation indefaultReporter.result(module.f.mjs:402-412), so following the proposed tasks leaves the CLI panic path untouched. Rewrite the design around those reporter normalization sites.AGENTS.md reference: AGENTS.md:L47-L51
Useful? React with 👍 / 👎.