-
-
Notifications
You must be signed in to change notification settings - Fork 6
emergent_testing: name a browser test the way fjs t does
#1738
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
2250a4a
03b3b06
a9f175c
dd3866a
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| - **BREAKING CHANGES:** `emergent_testing`: a browser test result gains a | ||
| required `name` — the test identity `fjs t` prints, built by the same | ||
| `fmtImport` function — and the page renders it, so both runners spell a test | ||
| identically. `renderBrowserReport` reads `name`, so a report built by hand | ||
| against the previous `{ module, path, ... }` shape renders `undefined` in | ||
| place of every identity; reports produced by `runBrowserProofs`, | ||
| `startBrowserTests` and `startBrowserTestSources` carry the field and are | ||
| unaffected. `module` and `path` are unchanged |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -18,7 +18,7 @@ | |
| * @import { _TestAndPath } from './types.ts' | ||
| */ | ||
|
|
||
| import { collectTests, fmtPath } from './module.f.mjs' | ||
| import { collectTests, fmtImport, fmtPath } from './module.f.mjs' | ||
|
|
||
| /** @type {(value: unknown) => string} */ | ||
| const text = value => { | ||
|
|
@@ -57,7 +57,25 @@ const errorDetails = error => { | |
| return [fallback, fallback] | ||
| } | ||
|
|
||
| /** @typedef {{ readonly module: string, readonly path: string, readonly status: string, readonly duration: number, readonly message?: string, readonly stack?: string }} _BrowserTestResult */ | ||
| /** | ||
| * `name` is the test's identity, and it is deliberately not built here: it comes | ||
| * from `fmtImport`, the same function `fjs t` prints its result lines with, so | ||
| * the two runners name a leaf identically — | ||
| * `import("./a.proof.f.mjs").proof.x()` in both. A page that invented its own | ||
| * spelling would produce reports that cannot be diffed against the console | ||
| * runner's, which is the visible half of the two runners having drifted apart. | ||
| * | ||
| * It is a field rather than something the renderer derives, because `module` | ||
| * and `path` cannot always be recombined into one: a module-level failure and a | ||
| * proof exported as a bare function both carry an empty `path`, and only the | ||
| * code that produced the result knows which it had. | ||
| * | ||
| * `path` stays for the consumers that already read it. It is now redundant with | ||
| * `name` for every leaf, and belongs in the report-shape decision this issue's | ||
| * todo tracks rather than in this change. | ||
| * | ||
| * @typedef {{ readonly module: string, readonly path: string, readonly name: string, readonly status: string, readonly duration: number, readonly message?: string, readonly stack?: string }} _BrowserTestResult | ||
| */ | ||
| /** @typedef {{ readonly status: string, readonly browser: string, readonly totals: { readonly tests: number, readonly passed: number, readonly failed: number }, readonly duration: number, readonly results: readonly _BrowserTestResult[] }} BrowserTestReport */ | ||
|
|
||
| /** | ||
|
|
@@ -182,11 +200,12 @@ const runPromise = (value, fulfilled, rejected) => { | |
| /** @type {(module: string, path: readonly (string | null)[], throws: boolean, fn: () => unknown, result: (result: _BrowserTestResult) => void) => Promise<readonly _BrowserTestResult[]>} */ | ||
| const runOne = (module, path, throws, fn, result) => { | ||
| const start = performance.now() | ||
| const name = fmtImport(module, path) | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When a proof or embedding page replaces AGENTS.md reference: AGENTS.md:L103-L106 Useful? React with 👍 / 👎. |
||
| /** @type {(value: unknown) => Promise<readonly _BrowserTestResult[]> | readonly _BrowserTestResult[]} */ | ||
| const passed = value => { | ||
| const duration = performance.now() - start | ||
| if (throws) { | ||
| const failure = { module, path: fmtPath(path), status: 'failed', duration, | ||
| const failure = { module, path: fmtPath(path), name, status: 'failed', duration, | ||
| message: 'Expected the proof to throw', stack: '' } | ||
| result(failure) | ||
| return [failure] | ||
|
|
@@ -205,7 +224,7 @@ const runOne = (module, path, throws, fn, result) => { | |
| return Promise.all(children.map(([childPath, child]) => | ||
| runOne(module, childPath, child.throws, child.fn, result) | ||
| )).then(results => { | ||
| const success = { module, path: fmtPath(path), status: 'passed', duration } | ||
| const success = { module, path: fmtPath(path), name, status: 'passed', duration } | ||
| result(success) | ||
| return [success, ...results.flat()] | ||
| }) | ||
|
|
@@ -214,12 +233,12 @@ const runOne = (module, path, throws, fn, result) => { | |
| const failed = error => { | ||
| const duration = performance.now() - start | ||
| if (throws) { | ||
| const success = { module, path: fmtPath(path), status: 'passed', duration } | ||
| const success = { module, path: fmtPath(path), name, status: 'passed', duration } | ||
| result(success) | ||
| return [success] | ||
| } | ||
| const [message, stack] = errorDetails(error) | ||
| const failure = { module, path: fmtPath(path), status: 'failed', duration, message, stack } | ||
| const failure = { module, path: fmtPath(path), name, status: 'failed', duration, message, stack } | ||
| result(failure) | ||
| return [failure] | ||
| } | ||
|
|
@@ -265,7 +284,7 @@ export const runBrowserProofs = (modules, result = () => undefined) => { | |
| /** @type {(module: string, error: unknown) => () => Promise<readonly _BrowserTestResult[]>} */ | ||
| const unreadable = (module, error) => () => { | ||
| const [message, stack] = errorDetails(error) | ||
| const failure = { module, path: '', status: 'failed', duration: 0, message, stack } | ||
| const failure = { module, path: '', name: module, status: 'failed', duration: 0, message, stack } | ||
| announce(failure) | ||
| return Promise.resolve([failure]) | ||
| } | ||
|
|
@@ -372,7 +391,7 @@ export const startBrowserTestSources = (root, sources, importer) => { | |
| return publish(root, Promise.resolve(reportOf('infrastructure-error', duration, | ||
| rejected.map(({ source, error }) => { | ||
| const [message, stack] = errorDetails(error) | ||
| return { module: source, path: '', status: 'failed', duration, message, stack } | ||
| return { module: source, path: '', name: source, status: 'failed', duration, message, stack } | ||
| })))) | ||
| } | ||
| return startBrowserTests(root, loadedModules.flatMap(module => | ||
|
|
@@ -431,7 +450,7 @@ const renderResult = (document, result) => { | |
| const item = document.createElement('li') | ||
| item.setAttribute('data-status', result.status) | ||
| const detail = result.status === 'failed' ? `: ${result.message}\n${result.stack}` : '' | ||
| item.textContent = `${result.status === 'passed' ? 'PASS' : 'FAIL'} ${result.module} ${result.path} (${result.duration.toFixed(1)} ms)${detail}` | ||
| item.textContent = `${result.status === 'passed' ? 'PASS' : 'FAIL'} ${result.name} (${result.duration.toFixed(1)} ms)${detail}` | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When the exported AGENTS.md reference: AGENTS.md:L91-L98 Useful? React with 👍 / 👎.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Correct, and taken as the second of the two options you named: documented as a breaking API change rather than given a fallback.
The fallback is the option I deliberately did not take. Retaining Generated by Claude Code |
||
| return item | ||
| } | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| ## An `index.html` for every module directory | ||
|
|
||
| **Priority:** P3 | ||
| **Status:** open | ||
|
|
||
| ### Problem | ||
|
|
||
| The generated website is one page. The repository it describes is a tree of | ||
| directories, most of which hold a `module.f.mjs`, its `types.ts`, a | ||
| `proof.f.mjs`, a `todo/` folder, and some subdirectories — and none of that is | ||
| reachable from the site. A reader who wants to know what `fjs/types/list` *is* | ||
| reads the source on GitHub; a reader who wants to know whether its proofs pass | ||
| runs the whole suite. Neither is a fact the website carries, and both are facts | ||
| it already has everything it needs to produce. | ||
|
|
||
| Browsing is the missing half. `fjs t` answers "did everything pass" and the | ||
| browser suite answers "does everything pass in a browser", but no view answers | ||
| "what is in this directory, and what does it prove?" — which is the question a | ||
| newcomer, and a maintainer looking at an unfamiliar corner, both start from. | ||
|
|
||
| ### Preliminary design | ||
|
|
||
| For every directory containing a `module.f.mjs` (and, after stage 2 of | ||
| [`migrate-typescript-to-mjs`](../../../todo/migrate-typescript-to-mjs.md), an | ||
| authored `module.f.js`), generate an `index.html` next to it in the output tree. | ||
| Each page is a catalog of that directory: | ||
|
|
||
| - **Files** — the modules, their `types.ts`, proofs and `README.md`, each linked | ||
| to a rendered source view where one exists. `README.md` conversion is already | ||
| on [generate-website](generate-website.md); this is a consumer of it. | ||
| - **Subdirectories** — linked to their own `index.html`, so the tree is | ||
| walkable in both directions. Include a breadcrumb back to the root. | ||
| - **Local proofs** — the tests this directory's modules contribute, named the | ||
| way both runners name them (`fmtImport`, `emergent_testing/module.f.mjs`), and | ||
| runnable *here*: the browser runner already takes a list of proof sources, so | ||
| a directory page is that same application with the manifest narrowed to this | ||
| directory. That is the interesting part of this issue — a per-directory page | ||
| is not a new runner, it is the existing one with a smaller list. | ||
| - **`todo/`** — the open issues filed against this directory, which are already | ||
| markdown next to the code and are the best available description of what is | ||
| unfinished in it. | ||
|
|
||
| Generation belongs in `fjs/website/module.f.mjs` as part of the same | ||
| `NodeProgram` that owns the rest of the build — the walk that discovers proof | ||
| sources today already visits every directory this needs, so this is a second | ||
| consumer of one traversal rather than a second traversal. See | ||
| [share-browser-console-runner](../../emergent_testing/todo/share-browser-console-runner.md) | ||
| for the preparation-program boundary this must respect: no npm script running an | ||
| impure helper as a second entry point, and any new filesystem capability | ||
| expressed as a Node effect with both interpretations proven. | ||
|
|
||
| ### Open questions | ||
|
|
||
| - **Does a page run its proofs on load, or on a `Run` click?** Per | ||
| [browser-test-controls](../../emergent_testing/todo/browser-test-controls.md) | ||
| a suite starts on an explicit action, and a directory page should not be an | ||
| exception just because it is small. | ||
| - **What does a directory with no `proof` export show?** An empty list is a | ||
| worse answer than saying that the modules here are proven from elsewhere, and | ||
| naming where. | ||
| - **How much of the source is rendered?** Linking to GitHub is free and | ||
| immediate; rendering source with highlighting is | ||
| [generate-website](generate-website.md)'s item and a larger change. A first | ||
| iteration can link out and still be useful. | ||
| - **Where does the output tree live**, relative to the isolated browser-test | ||
| application root that | ||
| [browser-testing](../../emergent_testing/todo/browser-testing.md) describes? | ||
| A directory page linking to modules is a page that serves source, which that | ||
| issue's application root deliberately does not do. These may be two output | ||
| trees rather than one. | ||
|
|
||
| ### Constraints | ||
|
|
||
| - The catalog is generated, never hand-maintained: a directory that gains a | ||
| module gains it on the page with no edit. | ||
| - A page must name a proof exactly as `fjs t` and the browser suite name it. | ||
| Three spellings of one test is the problem this repository has been removing. | ||
| - Do not build a second test runner. A directory page is the browser | ||
| application with a narrower manifest. | ||
| - No repository-wide index that has to be regenerated whenever any directory | ||
| changes; each page describes its own directory and links to its neighbours. | ||
|
|
||
| ### Tasks | ||
|
|
||
| - [ ] Generate an `index.html` per module directory, from the traversal the | ||
| website program already performs. | ||
| - [ ] List files, subdirectories, `todo/` issues, and a breadcrumb. | ||
| - [ ] Run the directory's own proofs on the page, through the existing browser | ||
| runner with a narrowed manifest. | ||
| - [ ] Decide the source-view question, and link out until it is answered. | ||
|
|
||
| ### Related | ||
|
|
||
| - [Generate website](generate-website.md) — README conversion, source | ||
| highlighting and `main.css`, all of which this page consumes. | ||
| - [Share the browser and console proof runners](../../emergent_testing/todo/share-browser-console-runner.md) | ||
| — the preparation-program boundary and the shared test name. | ||
| - [Browser testing](../../emergent_testing/todo/browser-testing.md) — the | ||
| application root and what it may serve. | ||
| - [Explicit browser test controls](../../emergent_testing/todo/browser-test-controls.md) | ||
| — a page does not auto-start a run. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When
fjs tis run from a supported subtree,loadModuleMapmakes its module keys relative to that working directory, whereas the browser manifest always supplies repository-root-relative sources. For example, the same leaf can therefore be namedimport("./module.f.mjs")...by the CLI andimport("./fjs/types/foo/module.f.mjs")...by the browser. Passing the rawmoduletofmtImportmeans the new field does not provide the promised cross-runner identity in this supported context, so report comparison, deduplication, or filtering still treats the same test as different; normalize both runners to a common module identifier before formatting.AGENTS.md reference: AGENTS.md:L60-L63
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The observation is right and the diagnosis is not — normalizing inside
fmtImport's callers would be the wrong fix.Reproduced. From the repository root:
and with
INIT_CWD=fjs/types/list:But note what that comparison is between: both lines are
fjs t. This is the CLI differing from itself across two invocations, not the CLI differing from the browser.loadModuleMapstrips theINIT_CWDprefix deliberately —loadModuleMapStripsInitCwdPrefixinfjs/dev/module.f.mjspins it — because a subtree run reports a subtree, and naming those leaves by a path the reader is not standing in would be the surprising behaviour.So a name embeds a module key, and a module key is relative to the root a run was given. Given the same key, the two runners produce the same name, which is exactly what this PR changes and what
nameMatchesTheConsoleRunnerpins. Given different roots they produce different keys — and would still do so after any normalization that did not also invent a canonical root, which is the part that cannot be assumed: the browser application root is not the repository root in the designemergent_testing/todo/browser-testing.mddescribes, so "canonicalize to the repository" would be picking one host's root and calling it universal.What is left of the finding is real and worth stating: two reports only compare when their roots agree, and nothing in the report declares its root. That is a report-shape question, not a naming one, and it now sits with the other open report-shape item (
path, which this PR makes redundant) inemergent_testing/todo/share-browser-console-runner.md. Fixing it here would widen the PR past the one thing it does.The PR body's "both runners spell a test identically" should be read as "for the same module key"; I have not changed the code.
Generated by Claude Code