Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions changelog/unreleased/1738.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
- **BREAKING CHANGES:** `emergent_testing`: a browser test result gains a
required `name` — the test identity `fjs t` prints, built by the same
`fmtImport` function — and the page renders it, so both runners spell a test
identically. `renderBrowserReport` reads `name`, so a report built by hand
against the previous `{ module, path, ... }` shape renders `undefined` in
place of every identity; reports produced by `runBrowserProofs`,
`startBrowserTests` and `startBrowserTestSources` carry the field and are
unaffected. `module` and `path` are unchanged
37 changes: 28 additions & 9 deletions fjs/emergent_testing/browser.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
* @import { _TestAndPath } from './types.ts'
*/

import { collectTests, fmtPath } from './module.f.mjs'
import { collectTests, fmtImport, fmtPath } from './module.f.mjs'

/** @type {(value: unknown) => string} */
const text = value => {
Expand Down Expand Up @@ -57,7 +57,25 @@ const errorDetails = error => {
return [fallback, fallback]
}

/** @typedef {{ readonly module: string, readonly path: string, readonly status: string, readonly duration: number, readonly message?: string, readonly stack?: string }} _BrowserTestResult */
/**
* `name` is the test's identity, and it is deliberately not built here: it comes
* from `fmtImport`, the same function `fjs t` prints its result lines with, so
* the two runners name a leaf identically —
* `import("./a.proof.f.mjs").proof.x()` in both. A page that invented its own
* spelling would produce reports that cannot be diffed against the console
* runner's, which is the visible half of the two runners having drifted apart.
*
* It is a field rather than something the renderer derives, because `module`
* and `path` cannot always be recombined into one: a module-level failure and a
* proof exported as a bare function both carry an empty `path`, and only the
* code that produced the result knows which it had.
*
* `path` stays for the consumers that already read it. It is now redundant with
* `name` for every leaf, and belongs in the report-shape decision this issue's
* todo tracks rather than in this change.
*
* @typedef {{ readonly module: string, readonly path: string, readonly name: string, readonly status: string, readonly duration: number, readonly message?: string, readonly stack?: string }} _BrowserTestResult
*/
/** @typedef {{ readonly status: string, readonly browser: string, readonly totals: { readonly tests: number, readonly passed: number, readonly failed: number }, readonly duration: number, readonly results: readonly _BrowserTestResult[] }} BrowserTestReport */

/**
Expand Down Expand Up @@ -182,11 +200,12 @@ const runPromise = (value, fulfilled, rejected) => {
/** @type {(module: string, path: readonly (string | null)[], throws: boolean, fn: () => unknown, result: (result: _BrowserTestResult) => void) => Promise<readonly _BrowserTestResult[]>} */
const runOne = (module, path, throws, fn, result) => {
const start = performance.now()
const name = fmtImport(module, path)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Canonicalize module paths before formatting names

When fjs t is run from a supported subtree, loadModuleMap makes its module keys relative to that working directory, whereas the browser manifest always supplies repository-root-relative sources. For example, the same leaf can therefore be named import("./module.f.mjs")... by the CLI and import("./fjs/types/foo/module.f.mjs")... by the browser. Passing the raw module to fmtImport means the new field does not provide the promised cross-runner identity in this supported context, so report comparison, deduplication, or filtering still treats the same test as different; normalize both runners to a common module identifier before formatting.

AGENTS.md reference: AGENTS.md:L60-L63

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The observation is right and the diagnosis is not — normalizing inside fmtImport's callers would be the wrong fix.

Reproduced. From the repository root:

import("./fjs/types/list/proof.f.mjs").proof.cycle[0](): ok

and with INIT_CWD=fjs/types/list:

import("./proof.f.mjs").proof.cycle[0](): ok

But note what that comparison is between: both lines are fjs t. This is the CLI differing from itself across two invocations, not the CLI differing from the browser. loadModuleMap strips the INIT_CWD prefix deliberately — loadModuleMapStripsInitCwdPrefix in fjs/dev/module.f.mjs pins it — because a subtree run reports a subtree, and naming those leaves by a path the reader is not standing in would be the surprising behaviour.

So a name embeds a module key, and a module key is relative to the root a run was given. Given the same key, the two runners produce the same name, which is exactly what this PR changes and what nameMatchesTheConsoleRunner pins. Given different roots they produce different keys — and would still do so after any normalization that did not also invent a canonical root, which is the part that cannot be assumed: the browser application root is not the repository root in the design emergent_testing/todo/browser-testing.md describes, so "canonicalize to the repository" would be picking one host's root and calling it universal.

What is left of the finding is real and worth stating: two reports only compare when their roots agree, and nothing in the report declares its root. That is a report-shape question, not a naming one, and it now sits with the other open report-shape item (path, which this PR makes redundant) in emergent_testing/todo/share-browser-console-runner.md. Fixing it here would widen the PR past the one thing it does.

The PR body's "both runners spell a test identically" should be read as "for the same module key"; I have not changed the code.


Generated by Claude Code

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Guard test-name formatting from proof-side mutation

When a proof or embedding page replaces JSON.stringify before a nested proof is scheduled, this synchronous fmtImport call throws before runOne installs its rejection handlers. Because fmtImport uses JSON.stringify, the exception rejects Promise.all and the entire runBrowserProofs call instead of producing a failed result, potentially leaving the page in running; the parent version resolves normally for the same identifier-only proof tree. Capture a safe formatter or handle name construction inside the protected execution path so this change does not introduce that regression.

AGENTS.md reference: AGENTS.md:L103-L106

Useful? React with 👍 / 👎.

/** @type {(value: unknown) => Promise<readonly _BrowserTestResult[]> | readonly _BrowserTestResult[]} */
const passed = value => {
const duration = performance.now() - start
if (throws) {
const failure = { module, path: fmtPath(path), status: 'failed', duration,
const failure = { module, path: fmtPath(path), name, status: 'failed', duration,
message: 'Expected the proof to throw', stack: '' }
result(failure)
return [failure]
Expand All @@ -205,7 +224,7 @@ const runOne = (module, path, throws, fn, result) => {
return Promise.all(children.map(([childPath, child]) =>
runOne(module, childPath, child.throws, child.fn, result)
)).then(results => {
const success = { module, path: fmtPath(path), status: 'passed', duration }
const success = { module, path: fmtPath(path), name, status: 'passed', duration }
result(success)
return [success, ...results.flat()]
})
Expand All @@ -214,12 +233,12 @@ const runOne = (module, path, throws, fn, result) => {
const failed = error => {
const duration = performance.now() - start
if (throws) {
const success = { module, path: fmtPath(path), status: 'passed', duration }
const success = { module, path: fmtPath(path), name, status: 'passed', duration }
result(success)
return [success]
}
const [message, stack] = errorDetails(error)
const failure = { module, path: fmtPath(path), status: 'failed', duration, message, stack }
const failure = { module, path: fmtPath(path), name, status: 'failed', duration, message, stack }
result(failure)
return [failure]
}
Expand Down Expand Up @@ -265,7 +284,7 @@ export const runBrowserProofs = (modules, result = () => undefined) => {
/** @type {(module: string, error: unknown) => () => Promise<readonly _BrowserTestResult[]>} */
const unreadable = (module, error) => () => {
const [message, stack] = errorDetails(error)
const failure = { module, path: '', status: 'failed', duration: 0, message, stack }
const failure = { module, path: '', name: module, status: 'failed', duration: 0, message, stack }
announce(failure)
return Promise.resolve([failure])
}
Expand Down Expand Up @@ -372,7 +391,7 @@ export const startBrowserTestSources = (root, sources, importer) => {
return publish(root, Promise.resolve(reportOf('infrastructure-error', duration,
rejected.map(({ source, error }) => {
const [message, stack] = errorDetails(error)
return { module: source, path: '', status: 'failed', duration, message, stack }
return { module: source, path: '', name: source, status: 'failed', duration, message, stack }
}))))
}
return startBrowserTests(root, loadedModules.flatMap(module =>
Expand Down Expand Up @@ -431,7 +450,7 @@ const renderResult = (document, result) => {
const item = document.createElement('li')
item.setAttribute('data-status', result.status)
const detail = result.status === 'failed' ? `: ${result.message}\n${result.stack}` : ''
item.textContent = `${result.status === 'passed' ? 'PASS' : 'FAIL'} ${result.module} ${result.path} (${result.duration.toFixed(1)} ms)${detail}`
item.textContent = `${result.status === 'passed' ? 'PASS' : 'FAIL'} ${result.name} (${result.duration.toFixed(1)} ms)${detail}`

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve reports that predate the name field

When the exported renderBrowserReport is called with a report created by an earlier version—or by an existing controller using the previous { module, path, ... } result shape—result.name is absent, so every row now renders as PASS undefined ... rather than displaying its identity. Either retain the old module/path rendering as a fallback or explicitly treat and document this as a breaking API change.

AGENTS.md reference: AGENTS.md:L91-L98

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, and taken as the second of the two options you named: documented as a breaking API change rather than given a fallback.

changelog/unreleased/1738.md now reads:

BREAKING CHANGES: emergent_testing: a browser test result gains a required namerenderBrowserReport reads name, so a report built by hand against the previous { module, path, … } shape renders undefined in place of every identity; reports produced by runBrowserProofs, startBrowserTests and startBrowserTestSources carry the field and are unaffected. module and path are unchanged

The fallback is the option I deliberately did not take. Retaining ${module} ${path} when name is absent would keep a second spelling of a test name alive inside the renderer, which is the exact thing this change exists to remove — a page that names a test one way for fresh reports and another way for older ones has two identities again, just conditionally. Every report the module produces carries the field, so no in-repository caller is affected, and the PR body now states the reasoning where a reader will look for it.


Generated by Claude Code

return item
}

Expand Down
23 changes: 21 additions & 2 deletions fjs/emergent_testing/browser/proof.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import { runInNewContext } from 'node:vm'

import { assert, assertEq, assertNotNullish, assertStructurallySame } from '../../asserts/module.f.mjs'
import { renderBrowserReport, runBrowserProofs, startBrowserTests, startBrowserTestSources } from '../browser.mjs'
import { fmtImport } from '../module.f.mjs'

/** @typedef {{ readonly tag: string, attributes: ReadonlyMap<string, string>, readonly ownerDocument: _Document, textContent: string, children: readonly _Element[], readonly setAttribute: (name: string, value: string) => void, readonly removeAttribute: (name: string) => void, readonly querySelector: (selector: string) => _Element | null, readonly replaceChildren: (...nodes: readonly _Element[]) => void, readonly append: (node: _Element) => void }} _Element */
/** @typedef {{ defaultView: _View | null, readonly createElement: (tag: string) => _Element }} _Document */
Expand Down Expand Up @@ -106,6 +107,24 @@ export const proof = {
const report = await run({ 'a.b': () => undefined })
assertEq(report.results[0]?.path, '["a.b"]')
},
// The page and `fjs t` must name a leaf identically, or two reports of the
// same suite cannot be compared. Asserting against `fmtImport` — the
// function the console runner prints its result lines with — is what makes
// that a shared fact rather than two spellings that happen to agree today.
nameMatchesTheConsoleRunner: async () => {
const report = await run({ nested: () => ({ child: () => undefined }) })
assertEq(report.results[0]?.name, fmtImport('proof', ['nested']))
assertEq(report.results[1]?.name, fmtImport('proof', ['nested', null, 'child']))
assertEq(report.results[1]?.name, 'import("proof").proof.nested().child()')
},
// A module that cannot be enumerated has no leaf to name, and an empty
// `path` does not distinguish it from a proof exported as a bare function.
// The module is what is known, so the module is the name.
unreadableModuleIsNamedByItsSource: async () => {
const report = await run(new Proxy({}, { ownKeys: () => { throw 'hostile' } }))
assertEq(report.status, 'failed')
assertEq(report.results[0]?.name, 'proof')
},
arbitraryThrow: async () => {
const report = await run({ fail: () => { throw Object.create(null) } })
assertEq(report.status, 'failed')
Expand Down Expand Up @@ -295,10 +314,10 @@ export const proof = {
browser: 'test',
totals: { tests: 1, passed: 1, failed: 0 },
duration: 1,
results: [{ module: 'm', path: '.t', status: 'passed', duration: 0.5 }],
results: [{ module: 'm', path: '.t', name: 'import("m").proof.t()', status: 'passed', duration: 0.5 }],
})
assertEq(p.summary.textContent, '1 passed, 0 failed (1.0 ms)')
assertEq(p.results.children[0]?.textContent, 'PASS m .t (0.5 ms)')
assertEq(p.results.children[0]?.textContent, 'PASS import("m").proof.t() (0.5 ms)')
},
sources: async () => {
const p = page()
Expand Down
50 changes: 48 additions & 2 deletions fjs/emergent_testing/todo/share-browser-console-runner.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,40 @@ it: the shared semantics first, with `fjs t` unchanged in behaviour and the
browser file only calling into it; the layout moves after; anything genuinely
new last, on its own.

### Steps

**One step per pull request.** The reverted attempt did the whole issue at once
— 2646 insertions and 1408 deletions across 35 files — and that is why its
arguments could not be separated: a question about scheduling became a question
about the port. Each step below stands on its own, leaves both runners working,
and is reviewable without the next one.

- [x] **1. One name function.** The page names a leaf with `fmtImport`, the
function `fjs t` prints its result lines with, so the two runners spell a
test identically. This is the smallest possible piece of the issue and
also its most visible symptom.
- [ ] **2. One `sandbox`.** Executing a proof body — the clock either side, the
`try`/`catch`, and the rule that only an actual `Promise` is awaited — is
the operation both runners must agree on exactly, and the one place where
they currently do not. Decide the cross-realm question
([imports, promises and realms](imports-promises-realms.md)) as part of
it, or record the decision, but do not let a port make it silently.
- [ ] **3. Common effects.** Move the host-independent operations (`all`,
`await`, `fetch`, `import`, `now`, `sandbox`) out of `effects/node` into a
shared module that `effects/node` re-exports unchanged, so nothing has to
move with them.
- [ ] **4. A browser interpreter** for exactly those operations, with no
scheduling policy of its own.
- [ ] **5. One reporter.** A normalized result the page and the terminal both
render, with no DOM and no terminal text in it.
- [ ] **6. One skeleton.** The page's proof-tree walk is deleted and the shared
traversal runs it.
- [ ] **7. The layout move**, and the website preparation program.

Steps 2 and 6 are the ones that change behaviour, so they are the ones to keep
smallest. Anything a step reveals goes to an issue and is fixed for both runners
later, never inside the step.

### Preliminary design

Share semantics, not host mechanics. The console runner should keep using the
Expand Down Expand Up @@ -193,6 +227,15 @@ are shared.
- Both runners must produce the same test name for the same leaf. This one is
not a host difference: nothing about a browser prevents it, and a divergence
here is the visible sign that the semantics underneath were never unified.
Note that a name embeds a *module key*, and a module key is relative to the
root a run was given: `fjs t` invoked in `fjs/types/list` names a leaf
`import("./proof.f.mjs")...` where the same leaf from the repository root is
`import("./fjs/types/list/proof.f.mjs")...`. That is `fjs t` differing from
itself across roots, not the two runners differing, and it is deliberate — a
subtree run reports a subtree. But two reports are only comparable when their
roots agree, and once the browser suite is a gate the question of which root a
report declares is worth settling. It belongs to the report shape, with
`path`.
- The skeleton never asks which host it is running on. Anything host-specific is
a part it calls; anything it cannot express through a part is a missing
extension point, not a special case.
Expand Down Expand Up @@ -229,8 +272,11 @@ are shared.
of them.
- [ ] Make the existing `collectTests`/path behavior the single source of truth
for console and browser execution.
- [ ] Share the test-name format, and prove both runners name the same leaf
identically.
- [x] Share the test-name format, and prove both runners name the same leaf
identically. The browser report carries a `name` built by `fmtImport`, and
`nameMatchesTheConsoleRunner` pins it to that function rather than to a
spelling. Its `path` field is now redundant with `name` for every leaf and
should go when the report shape is decided.
- [ ] Define normalized leaf, progress, infrastructure-error, totals, and report
values without terminal or DOM fields.
- [ ] Decide whether browser import/time/yield/publication justify
Expand Down
101 changes: 101 additions & 0 deletions fjs/website/todo/directory-index-pages.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
## An `index.html` for every module directory

**Priority:** P3
**Status:** open

### Problem

The generated website is one page. The repository it describes is a tree of
directories, most of which hold a `module.f.mjs`, its `types.ts`, a
`proof.f.mjs`, a `todo/` folder, and some subdirectories — and none of that is
reachable from the site. A reader who wants to know what `fjs/types/list` *is*
reads the source on GitHub; a reader who wants to know whether its proofs pass
runs the whole suite. Neither is a fact the website carries, and both are facts
it already has everything it needs to produce.

Browsing is the missing half. `fjs t` answers "did everything pass" and the
browser suite answers "does everything pass in a browser", but no view answers
"what is in this directory, and what does it prove?" — which is the question a
newcomer, and a maintainer looking at an unfamiliar corner, both start from.

### Preliminary design

For every directory containing a `module.f.mjs` (and, after stage 2 of
[`migrate-typescript-to-mjs`](../../../todo/migrate-typescript-to-mjs.md), an
authored `module.f.js`), generate an `index.html` next to it in the output tree.
Each page is a catalog of that directory:

- **Files** — the modules, their `types.ts`, proofs and `README.md`, each linked
to a rendered source view where one exists. `README.md` conversion is already
on [generate-website](generate-website.md); this is a consumer of it.
- **Subdirectories** — linked to their own `index.html`, so the tree is
walkable in both directions. Include a breadcrumb back to the root.
- **Local proofs** — the tests this directory's modules contribute, named the
way both runners name them (`fmtImport`, `emergent_testing/module.f.mjs`), and
runnable *here*: the browser runner already takes a list of proof sources, so
a directory page is that same application with the manifest narrowed to this
directory. That is the interesting part of this issue — a per-directory page
is not a new runner, it is the existing one with a smaller list.
- **`todo/`** — the open issues filed against this directory, which are already
markdown next to the code and are the best available description of what is
unfinished in it.

Generation belongs in `fjs/website/module.f.mjs` as part of the same
`NodeProgram` that owns the rest of the build — the walk that discovers proof
sources today already visits every directory this needs, so this is a second
consumer of one traversal rather than a second traversal. See
[share-browser-console-runner](../../emergent_testing/todo/share-browser-console-runner.md)
for the preparation-program boundary this must respect: no npm script running an
impure helper as a second entry point, and any new filesystem capability
expressed as a Node effect with both interpretations proven.

### Open questions

- **Does a page run its proofs on load, or on a `Run` click?** Per
[browser-test-controls](../../emergent_testing/todo/browser-test-controls.md)
a suite starts on an explicit action, and a directory page should not be an
exception just because it is small.
- **What does a directory with no `proof` export show?** An empty list is a
worse answer than saying that the modules here are proven from elsewhere, and
naming where.
- **How much of the source is rendered?** Linking to GitHub is free and
immediate; rendering source with highlighting is
[generate-website](generate-website.md)'s item and a larger change. A first
iteration can link out and still be useful.
- **Where does the output tree live**, relative to the isolated browser-test
application root that
[browser-testing](../../emergent_testing/todo/browser-testing.md) describes?
A directory page linking to modules is a page that serves source, which that
issue's application root deliberately does not do. These may be two output
trees rather than one.

### Constraints

- The catalog is generated, never hand-maintained: a directory that gains a
module gains it on the page with no edit.
- A page must name a proof exactly as `fjs t` and the browser suite name it.
Three spellings of one test is the problem this repository has been removing.
- Do not build a second test runner. A directory page is the browser
application with a narrower manifest.
- No repository-wide index that has to be regenerated whenever any directory
changes; each page describes its own directory and links to its neighbours.

### Tasks

- [ ] Generate an `index.html` per module directory, from the traversal the
website program already performs.
- [ ] List files, subdirectories, `todo/` issues, and a breadcrumb.
- [ ] Run the directory's own proofs on the page, through the existing browser
runner with a narrowed manifest.
- [ ] Decide the source-view question, and link out until it is answered.

### Related

- [Generate website](generate-website.md) — README conversion, source
highlighting and `main.css`, all of which this page consumes.
- [Share the browser and console proof runners](../../emergent_testing/todo/share-browser-console-runner.md)
— the preparation-program boundary and the shared test name.
- [Browser testing](../../emergent_testing/todo/browser-testing.md) — the
application root and what it may serve.
- [Explicit browser test controls](../../emergent_testing/todo/browser-test-controls.md)
— a page does not auto-start a run.
3 changes: 3 additions & 0 deletions fjs/website/todo/generate-website.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@
- [ ] Source code highlighting
- [ ] One `main.css`
- [ ] Convention for `page.f.mjs` — generates a demo webpage for the module in the same directory
- [ ] An `index.html` per module directory, cataloguing its files,
subdirectories, `todo/` issues and local proofs — see
[directory-index-pages](directory-index-pages.md)
- [x] Browser test runner and proof-result UI
- [ ] Move browser-manifest preparation into the website `NodeProgram` through
Node effects, as designed in
Expand Down
Loading