Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
01f29ad
emergent_testing: one proof runner for `fjs t` and the browser
sergey-shandar Aug 27, 2026
62ba9d1
changelog: entry for #1737
sergey-shandar Aug 27, 2026
b457587
Merge branch 'main' into claude/todo-unification-runners-7wvd1q
sergey-shandar Aug 27, 2026
0aac225
emergent_testing/browser: yield between batches, and never end a run …
sergey-shandar Aug 27, 2026
dc10dcf
effects: share the impure sandbox, and restore the browser proofs rev…
sergey-shandar Aug 27, 2026
57e295b
emergent_testing/browser: make the panic guard total, and stop diagno…
sergey-shandar Aug 27, 2026
68fc983
emergent_testing/todo: design the `catch` operation that guards the t…
sergey-shandar Aug 27, 2026
505766d
emergent_testing: one spelling for a test's name, and the three TODOs…
sergey-shandar Aug 27, 2026
569636b
emergent_testing/todo: put the browser batching on probation rather t…
sergey-shandar Aug 27, 2026
4923ddb
emergent_testing/todo: measure the browser scheduling instead of argu…
sergey-shandar Aug 27, 2026
cbc4454
effects/browser: `all` must start every child before it awaits any
sergey-shandar Aug 27, 2026
0773613
emergent_testing/todo: record the lost loader stack
sergey-shandar Aug 27, 2026
401f191
effects/browser: delete the batch size; yield after every launch
sergey-shandar Aug 27, 2026
5a74d4d
emergent_testing/todo: report a test's name before running it
sergey-shandar Aug 27, 2026
09bac1f
emergent_testing/browser: assert the yield on the queue all posts to
claude Aug 27, 2026
6f5dd59
DESIGN: follow the example when porting a capability to a second context
claude Aug 27, 2026
565f4ce
Revert the shared runner; keep what it taught
claude Aug 27, 2026
fe112e8
DESIGN: state the sharing cycle as five steps, ending in "solve for b…
claude Aug 27, 2026
b9b273c
DESIGN: the shared thing is the skeleton, not just the outcome
claude Aug 27, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 70 additions & 0 deletions DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,8 +107,78 @@ on top of the weaker design.
belongs in `fjs/path`, not inline in a loader). First search for an appropriate
existing module; create a new one only if no good fit exists. This is different
from DRY extraction: it is always appropriate.
- **Follow the example** — one skeleton for every context; differences live in
the parts it calls, and improvements go into the skeleton so everyone gets
them. See below.
- **Avoid side effects and mutability.**

### Follow the example

When a capability already exists somewhere in the repository and is being
brought to a second context — another host, another backend, another runner —
**the existing one is the specification.**

What is shared is the **skeleton**: the control flow, the order of operations,
the decisions and their names — the shape of the whole thing. Every context runs
that same skeleton. Where a context differs, it differs by supplying a different
**part** that the skeleton calls out to, at a place the skeleton names. It does
not differ by having a skeleton of its own.

So there are exactly two ways to accommodate a context, and both are additive:

- **Adjust that context's part.** A browser writes rows into a DOM where a
terminal writes lines to stdout; those are two implementations of one named
part, and the skeleton above them cannot tell which it has.
- **Improve the skeleton, for everyone.** If what the new context needs is
something the skeleton should have had, put it there. Every context gets it,
and that is a feature of the change rather than a side effect to apologize
for.

There is no third way. A branch inside the skeleton that asks which host it is
running on is a fork wearing a shared name, and it is worse than two honest
implementations, because nothing about the shared name signals the difference. A
context that cannot be served by any existing part means the skeleton is missing
an extension point: add the point — one more named part that every context then
supplies — rather than a special case.

The order of work follows from that:

1. **Share the skeleton.** Take the existing implementation as the core, with
its behaviour unchanged.
2. **Adjust the parts** the new context genuinely requires, or extend the
skeleton so it can express what the new context needs.
3. **Document every difference that remains,** at the part where it is made.
4. **Open an issue for each problem the port revealed,** rather than fixing it
inside the port.
5. **Solve each issue in the skeleton or in every part at once,** so the
contexts stay in sync.

The parts worth stating outright:

**Differences are allowed; undocumented differences are not.** The goal is not
one identical behaviour — a browser has no stdout and a terminal has no DOM, and
pretending otherwise invents a host that does not exist. The goal is that every
difference lives in a named part, is deliberate, and is traceable to something
the host forced. "This context could do better here" is not such a reason: that
is an improvement, and an improvement belongs in the skeleton, where everyone
gets it.

**Solve it for every context, or for none.** Once an issue from step 4 is picked
up, the fix lands everywhere in the same change. A fix in one context only is how
the contexts drift back apart, and it hides the finding from the place that has
had the defect longest — usually the older one.

**The example may be simple for a reason.** What looks like a gap from inside
the new context is often a decision made in the old one. Copy it first; if it
turns out to be wrong, it is wrong in both places and worth an issue that says
so.

**Keep the port separate from everything it inspires.** Land the sharing change
on its own, with behaviour unchanged. Anything new — a different scheduling
policy, a better measurement, an extra guard — is its own change afterwards.
Combined, they cannot be reviewed: an argument about the new idea becomes an
argument about the port.

### Exception to DRY: performance measurement

Time measurement must capture immediately after an operation completes to avoid
Expand Down
113 changes: 113 additions & 0 deletions fjs/emergent_testing/todo/hostile-proof-values.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
## Hostile thrown values and cross-realm promises kill a run

**Priority:** P3
**Status:** open

### Problem

The browser runner (`../browser.mjs`) defends against two things `fjs t` does
not, and neither is reachable from ordinary FunctionalScript. That asymmetry is
the point of this file: when the two runners are unified
([share the browser and console proof runners](share-browser-console-runner.md)),
the shared core has to have *one* answer for each of them, decided rather than
inherited twice. `fjs t` is the reference, so the honest reading is that these
are gaps in `fjs t` which the browser happened to cover — and closing them in
the shared core is the way to keep that coverage instead of losing it to a port.

**A value that resists being read is not attributed to the test that produced
it.** Two functions in the shared core read user-supplied values without a
guard: the `collectTests` traversal enumerates a returned proof tree, and
`errorDetails` reads `message`/`stack` and calls `String` on a thrown value. A
Comment on lines +17 to +20

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Point the guard at the CLI normalization sites

For hostile values thrown under fjs t, this problem statement identifies errorDetails as an unguarded shared-core reader, but at this commit that helper is browser-only and already guards both property access and String conversion (browser.mjs:24-57). The actual CLI coercions are the unguarded String(v) and template interpolation in defaultReporter.result (module.f.mjs:402-412), so following the proposed tasks leaves the CLI panic path untouched. Rewrite the design around those reporter normalization sites.

AGENTS.md reference: AGENTS.md:L47-L51

Useful? React with 👍 / 👎.

throwing accessor, a revoked `Proxy`, or a hostile `toString` panics through
either, and there is no `try`/`catch` in FunctionalScript for the core to catch
it with. `fjs t` ends with a stack trace and no summary; the browser runner
today loses one test and carries on. What is missing from the core is
*attribution*: naming the leaf whose value could not be read, and continuing
with the rest. Whichever runner ends up on top of it, a page left in `running`
or a process that exits with no summary is the outcome an automated controller
cannot act on.

**A promise from another realm is not awaited.** `fjs t`'s `sandbox` asks `p
instanceof Promise`, which is false for a promise built in an iframe, a worker,
or a `node:vm` context. Such a value is walked as an ordinary proof tree
instead, so a *rejected* cross-realm promise is reported as a pass. The browser
runner carries `Symbol.species` machinery against this, which is a second answer
to the same question and is studied in
[imports, promises and realms](imports-promises-realms.md). The obvious repair —
brand-checking with `Object.prototype.toString` — is not one: the tag is
settable through `Symbol.toStringTag`, and an object carrying a `then` proof
would then be assimilated, breaking the rule that only actual promises are
asynchronous values.

### Design: a `catch` operation

Reading a user value belongs to the *operation*, not to the shared core, which
is what makes one fix serve every runner. Once the two runners share a core,
guarding the traversal once covers `fjs t` and the browser together — which is
an argument for doing this *with* the sharing change rather than before it.

**`sandbox` cannot hold it, and the reason is not the one it looks like.**
Timing is not the obstacle: the sub-tree walk in `runModule` happens *after* the
runner has resolved the leaf's promise, so `sandbox(() => collectTests(path,
false, r))` would run a pure synchronous thunk over an already-settled value.
The obstacle is the **virtual runner**. Its `sandbox` is a deliberate
pass-through — `f => state => [state, ok(f())]`, with the fixture returning the
`SandboxResult` it wants reported — because `../../effects/node/virtual` is
`.f.mjs` and FunctionalScript has no `try`/`catch` to implement a real one with.
Routing the traversal through `sandbox` would hand that handler a thunk
answering `_TestAndPath[]`, which it would cast to `SandboxResult` and every
fixture in `../proof.f.mjs` would break.

So add a second, honest operation beside it:

```ts
export type Catch = readonly['catch', <T>(f: () => T) => OpResult<Result<T, unknown>>]
```

"Run this pure thunk; a throw is the `error` branch." It carries no clock and no
fixture convention, so each runner implements it truthfully:

- the real Node runner, and whatever browser interpreter the sharing change
produces: `tryCatch(f)`, one line each, from `types/result/module.mjs`.
- `effects/node/virtual/module.f.mjs`: `ok(ok(f()))` — a pure runner still
cannot catch, and a hostile fixture still panics there, which is the same
bargain `sandbox` already makes. Virtual proofs use benign fixtures.

`walk` then reads a sub-tree through `catch` and, on the `error` branch, reports
one failed result at that path instead of panicking — which is what restores
`exportedTreeThrows` / `returnedTreeThrows`, and gives `fjs t` a behaviour it
never had. `errorDetails` gets the same treatment at its one call site.
Comment on lines +76 to +79

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Make the proposed normalization fallback total

When errorDetails inspects a self-throwing proxy, wrapping that call in catch only produces another error containing the same hostile value; attempting to normalize that error repeats the trap, so the promised failed result still cannot be constructed. The fresh evidence in this documentation commit is that the proposed design records only “the same treatment” and never captures the trap-free terminal fallback learned from the earlier self-throwing-proxy fix. Specify a constant fallback that performs no operations on the caught value, and add that hostile case to the tasks, before this design is implemented.

AGENTS.md reference: AGENTS.md:L47-L51

Useful? React with 👍 / 👎.


The work is roughly: the operation and its constructor beside `sandbox`, one
handler in each runner, the `CommandSet` entries, the `walk` change and its new
result shape, and the mock maps in the affected proofs.

**The brand check** for cross-realm promises is not designed here. It belongs
with the two mechanisms it keeps being confused with — a module namespace
adopting a `then`, and a proof tree refusing to — which are studied together in
[imports, promises and realms](imports-promises-realms.md).

### Tasks

- [ ] Add the `catch` operation, its constructor, and a handler in each of the
Node, browser and virtual runners.
- [ ] Read sub-trees through it in `walk`, reporting an unreadable tree as one
failed result at its path rather than a panic.
- [ ] Prove an unreadable exported tree and an unreadable returned tree, for
`fjs t` as well as the browser — the browser has versions of these today
and `fjs t` has none.
- [ ] Read a thrown value through it at `errorDetails`' call site.

### Constraints

- Whatever is added must apply to `fjs t` and to the browser runner alike. A
defense in one runner only is the thing to avoid: it is how the two came to
mean different things in the first place.
- An object carrying a `then` proof property must stay an ordinary proof tree.

### Related

- [Imports, promises and realms](imports-promises-realms.md) — where the
cross-realm brand check is studied.
- [Browser testing](browser-testing.md)
- [Test-runner behavior](661-test-runner-behavior.md)
78 changes: 78 additions & 0 deletions fjs/emergent_testing/todo/imports-promises-realms.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
## Investigate imports, promises and realms

**Priority:** P3
**Status:** open — investigation, not yet actionable

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use a canonical todo status

The repository's todo format restricts Status to exact values such as open, but this value appends an explanation and therefore no longer matches any allowed status. Keep the field as **Status:** open and move “investigation, not yet actionable” into the issue body so status searches and tooling can classify it consistently.

AGENTS.md reference: AGENTS.md:L55-L56

Useful? React with 👍 / 👎.


### Problem

Three mechanisms meet in the runner, none of them is written down as a rule, and
the code where they meet reads as a pile of special cases rather than a design.
They are separate mechanisms that happen to interact, and the interaction is
what nobody has stated:

**A module namespace object is a thenable.** `import()` resolves by *adopting*
what a module exports, so a module exporting a function named `then` corrupts
its own dynamic import. That is why exporting `then` from a proof module is
forbidden ([`spec/todo/3240-export.md`](../../../spec/todo/3240-export.md)) —
but the rule lives in a spec issue and a README paragraph, and nothing checks
it. The proof discovery in `../../dev/module.f.mjs` imports whatever it finds.

**A proof tree is not a thenable, even when it has a `then`.** The runner's rule
is that only an actual `Promise` is an asynchronous value, so `{ then: f }`
returned from a proof is a sub-tree with a test called `then` in it. This is the
opposite reading of the same property name, one layer down, and both readings
are correct in their own layer. Nothing says so in one place.

**`instanceof Promise` is realm-local.** A promise built in an iframe, a worker
or a `node:vm` context is not `instanceof Promise` here, so under `fjs t` it is
walked as a proof tree and a *rejected* one is reported as a pass. The browser
runner defends against this with `Symbol.species` shadowing and an intrinsic
`then` — about 150 lines (`../browser.mjs`, `../browser/species.proof.mjs`) that
read as a magic mess and are, today, the only place the exposure is covered. So
the two runners answer this question differently, and
[sharing them](share-browser-console-runner.md) forces a single answer: keep the
machinery, replace it with something statable, or accept `fjs t`'s exposure
knowingly. Deciding that by default, inside a port, is how the coverage gets
lost without anyone choosing to lose it.

The three are usually discussed one at a time, which is why the interaction
keeps being rediscovered: the thing that makes a namespace dangerous (`then` is
adopted) is the thing the runner deliberately refuses to do (`then` is a name),
and the check that separates them (`instanceof`) is the one that does not
survive a realm boundary.

### What to investigate

This is a study, not a design. It is worth doing before
[browser-testing](browser-testing.md) puts proofs in iframes or workers, because
that is the point at which cross-realm promises stop being hypothetical.

- **State the layering.** One document saying which layer adopts a `then` and
which layer refuses to, and why both are right. Until that exists, every fix
to one looks like a bug in the other.
- **Find a brand check that survives a realm and cannot be forged.**
`Object.prototype.toString` is forgeable through `Symbol.toStringTag`.
`Promise.resolve(p) === p` against the value's own constructor is a candidate.
Whatever is chosen must be one function every interpreter calls.
- **Decide whether the runner should see namespace objects at all.** If
discovery handed the runner a plain record of proofs rather than the module
namespace, the `then` export hazard would not reach it — and the `then`-export
ban could become a check rather than a convention.
- **Establish what the 150 lines actually buy**, from the proofs that cover them
(`../browser/species.proof.mjs`), so that whatever replaces them is measured
against the same cases rather than against a memory — and so that removing
them, if that is the answer, is a decision with a list attached.

### Constraints

- An object carrying a `then` proof property must stay an ordinary proof tree.
- Whatever is added must apply to every runner. A defence in one host only is
the state this is trying to leave.

### Related

- [Hostile proof values](hostile-proof-values.md) — the cross-realm promise
exposure, and the traversal guard it shares a cause with.
- [Browser testing](browser-testing.md) — iframes and workers.
- [`spec/todo/3240-export.md`](../../../spec/todo/3240-export.md) — the `then`
export ban.
79 changes: 79 additions & 0 deletions fjs/emergent_testing/todo/report-before-running.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
## Report a test's name before running it, not only after

**Priority:** P2
**Status:** open

### Problem

Every runner reports a test only once it has finished. `fjs t` writes
`import("./a.proof.f.mjs").proof.x(): ok, 0.3 ms` after the fact, and the
browser page appends `PASS import("a").proof.x() (0.3 ms)` the same way. A test
that is *running* is invisible.

Three things follow from that, and the third is the one that matters:

- **A slow test looks like a hung runner.** Nothing distinguishes "this proof
has been going for ten seconds" from "the runner stopped", so the only way to
find the slow one is to wait for it to finish and read the duration.
- **Progress is a count, not a place.** The browser page says "1247 tests
completed…" while a reader wants to know *which* one it is on.
- **A crash loses the one fact worth having.** When a proof takes the process
down — a panic through the shared traversal, an out-of-memory, a stack
overflow, a runner bug — the last line printed is the last test that
*succeeded*, and the one that actually broke is never named. That is exactly
the case where a name is worth more than a result, and it is the case where
the current design has none.

No reporter has an event for it: `result` is called with a `SandboxResult`, so it
cannot be called before there is one.

### Preliminary design

Add a `start` (or `begin`) event to the reporter, called with the file and path
before the leaf is sandboxed, and let each host decide what to do with it:

- **`fjs t`** prints the name, then completes the line with `ok`/`error` and the
duration when the result lands — the standard runner shape, in the format it
already prints. Interleaving is the thing to get right: leaves
run concurrently, so a half-written line cannot be left open across another
test's output. Either the name and its outcome are one deferred line with the
name shown live elsewhere, or output is a two-column log that names the start
and closes it by identifier.
- **The browser page** renders a row in a pending state and settles it in place,
which is the same list it renders now with one more state per row.
- **A result type** may not need to change at all: a start is an event, not a
result. Whether the reporter grows a sibling operation or its existing one
gains a status is part of the design.

The reporter change is small; the interleaving question is the real one, and
it is the same question in both hosts, which is an argument for settling it in
the shared core rather than twice.

### Constraints

- A start event must not cost a `sandbox` call or a clock read of its own: the
duration reported is still the sandboxed one.
- Concurrency stays. Naming a test before running it must not serialize the
suite to keep the output tidy.
- Whatever is emitted has to be as useful to an automated consumer as to a
reader — a start with no matching result is precisely the signal a crashed
run leaves behind, and a controller should be able to read it.
- The start event lands in both runners in the same change. Their output differs
— a terminal line and a DOM row — but a runner that names a running test and
one that does not are two different tools.

### Tasks

- [ ] Add the start event to the reporter and call it before the
leaf is sandboxed.
- [ ] Decide the terminal format for concurrent output, and prove it.
- [ ] Render a pending row in the browser page and settle it in place.
- [ ] Prove that a run killed mid-test leaves the running test's name behind.

### Related

- [Share the browser and console proof runners](share-browser-console-runner.md)
— reporting is one of the things each host still does its own way, and this
is the same question twice until they share a reporter.
- [Hostile proof values](hostile-proof-values.md) — the crash case this would
make diagnosable, where today the run ends with no summary and no name.
Loading
Loading