Skip to content

perf: memoize URL and route resolution in link codecs - #1951

Closed
spa5k wants to merge 4 commits into
middleapi:mainfrom
spa5k:perf/client-codec-memoization
Closed

perf: memoize URL and route resolution in link codecs#1951
spa5k wants to merge 4 commits into
middleapi:mainfrom
spa5k:perf/client-codec-memoization

Conversation

@spa5k

@spa5k spa5k commented Aug 25, 2026

Copy link
Copy Markdown

What this PR does

The client made the same URL and route data again for each call. This PR caches that data. Profiling showed about 8.6 percent of each call in this work.

What changed

  1. pathToHttpPath caches by path array identity and validates a snapshot of its segments. Stable procedure paths hit the fast path. A mutated array is recomputed correctly.
  2. Both link codecs cache the parsed base URL. The cache holds one entry. The key is the URL value. Callers that change the URL per call still get correct results.
  3. The OpenAPI codec caches the resolved route. The cache holds the meta, the method, the path template, and the dynamic params. The old code walked the contract tree two times per call. The new code walks it one time. Lazy contracts load one time. A failed load is not cached. Synchronous loader failures can retry, and route resolution snapshots the path before awaiting.
  4. StandardLink.call reads the tracer config one time. When no tracer is set, the code skips the span names and the three runWithSpan wrappers. Tracing behavior does not change when a tracer is set.
  5. The client builds the interceptors array one time at creation. Before, the code built it for each call.
  6. The untraced fast path still returns a rejected Promise when an interceptor throws synchronously.

How much faster

Test setup: Apple M4, Node v25.9.0, Vitest v4.1.11, tracing off. The base is current main (29917766).

Test Before After Change
OpenAPI codec, path depth 10 1.83µs 0.39µs 4.65x faster
OpenAPI codec, path depth 20 2.66µs 0.49µs 5.39x faster
OpenAPI codec, 1000 procedures 0.87µs 0.40µs 2.16x faster
RPC codec, path depth 10 0.83µs 0.37µs 2.20x faster
RPC codec, path depth 20 1.34µs 0.39µs 3.44x faster
OpenAPI codec, 5 dynamic params 2.51µs 1.79µs 1.40x faster

The codec cost now stays almost flat when the path gets deeper. End-to-end timings vary more, so they are not used as headline results.

Known trade-off

The path cache uses array identity and validates a segment snapshot. A caller that makes a new array for each call does not hit the cache. That caller pays the WeakMap lookup on top of the encoding. At depth 20, a fresh array is about 23 percent slower than main. Clients made with createORPCClient reuse path arrays. That case is 3.44 times faster.

Benchmark scenarios

The codec scenarios live in the existing benchmark files. This PR adds no new files.

  • rpc-link-handler.bench.ts: codec path encoding at depth 10 and 20. One scenario uses a fresh array per call. This is the cache-miss worst case.
  • openapi-link-handler.bench.ts: codec route resolution at depth 10 and 20, a router with 1000 procedures, 5 dynamic path params, and a GET query with 50 params.

Behavior notes

  • The route cache is correct when the contract does not change after creation. The server matchers make the same assumption.
  • When no tracer is set, the client does not wrap async iterator inputs. The wrap exists only for spans. The iterator behavior is the same.
  • Changes to a client's interceptors array after creation are not seen by later calls. Before, the code read the array again for each call.

Tests

  • Focused client, shared, and OpenAPI tests: 187 passed.
  • Both benchmark files load and run against clean main and this branch.
  • Bun 1.4.0 coverage suite: 97 passed and 31 skipped.
  • Regression tests cover mutable path arrays, synchronous lazy-loader retry, path mutation during lazy loading, and synchronous interceptor errors preserving the Promise contract.
  • eslint and package diagnostics pass with no errors.

@pkg-pr-new

pkg-pr-new Bot commented Aug 25, 2026

Copy link
Copy Markdown
More templates

@orpc/ai-sdk

npm i https://pkg.pr.new/@orpc/ai-sdk@1951

@orpc/arktype

npm i https://pkg.pr.new/@orpc/arktype@1951

@orpc/bun

npm i https://pkg.pr.new/@orpc/bun@1951

@orpc/client

npm i https://pkg.pr.new/@orpc/client@1951

@orpc/cloudflare

npm i https://pkg.pr.new/@orpc/cloudflare@1951

@orpc/contract

npm i https://pkg.pr.new/@orpc/contract@1951

@orpc/experimental-effect

npm i https://pkg.pr.new/@orpc/experimental-effect@1951

@orpc/evlog

npm i https://pkg.pr.new/@orpc/evlog@1951

@orpc/hibernation

npm i https://pkg.pr.new/@orpc/hibernation@1951

@orpc/json-schema

npm i https://pkg.pr.new/@orpc/json-schema@1951

@orpc/experimental-msw

npm i https://pkg.pr.new/@orpc/experimental-msw@1951

@orpc/nest

npm i https://pkg.pr.new/@orpc/nest@1951

@orpc/next

npm i https://pkg.pr.new/@orpc/next@1951

@orpc/node

npm i https://pkg.pr.new/@orpc/node@1951

@orpc/openapi

npm i https://pkg.pr.new/@orpc/openapi@1951

@orpc/opentelemetry

npm i https://pkg.pr.new/@orpc/opentelemetry@1951

@orpc/pinia-colada

npm i https://pkg.pr.new/@orpc/pinia-colada@1951

@orpc/pino

npm i https://pkg.pr.new/@orpc/pino@1951

@orpc/publisher

npm i https://pkg.pr.new/@orpc/publisher@1951

@orpc/ratelimit

npm i https://pkg.pr.new/@orpc/ratelimit@1951

@orpc/server

npm i https://pkg.pr.new/@orpc/server@1951

@orpc/shared

npm i https://pkg.pr.new/@orpc/shared@1951

@orpc/swr

npm i https://pkg.pr.new/@orpc/swr@1951

@orpc/tanstack-query

npm i https://pkg.pr.new/@orpc/tanstack-query@1951

@orpc/trpc

npm i https://pkg.pr.new/@orpc/trpc@1951

@orpc/valibot

npm i https://pkg.pr.new/@orpc/valibot@1951

@orpc/zod

npm i https://pkg.pr.new/@orpc/zod@1951

commit: 06effe7

@codecov

codecov Bot commented Aug 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@codspeed-hq

codspeed-hq Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will improve performance by 10.97%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 29 untouched benchmarks
🆕 8 new benchmarks

Performance Changes

Benchmark BASE HEAD Efficiency
octet stream 912.2 µs 822 µs +10.97%
🆕 encodeInput GET with 50 query params N/A 760.5 µs N/A
🆕 encodeInput with 5 dynamic path params N/A 235.4 µs N/A
🆕 encodeInput at path depth 10 N/A 43.4 µs N/A
🆕 encodeInput at path depth 20 N/A 45.9 µs N/A
🆕 encodeInput on a 1000-procedure router N/A 42.6 µs N/A
🆕 encodeInput at path depth 10 N/A 37.9 µs N/A
🆕 encodeInput at path depth 20 N/A 39 µs N/A
🆕 encodeInput at path depth 20 (fresh array per call) N/A 109.5 µs N/A

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing spa5k:perf/client-codec-memoization (06effe7) with main (2991776)

Open in CodSpeed

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ No critical issues — one minor suggestion inline.

This PR is stacked on #1950; I reviewed only the last two commits (41e3985 perf change and f2a3e54 bench fold). The memoization is well-engineered and well-tested: the WeakMap identity cache in pathToHttpPath, the single-slot base-url parse caches, and the OpenAPILinkCodec route memoization (with in-flight dedupe via routePromises cleared in finally, so failed resolves are retried) are all correct. The dotted-vs-nested cache-key test genuinely pins the stringifyJSON keying, and the lazy-load tests meaningfully exercise the dedupe/not-cached semantics. I ran the affected suites (191 tests) — all pass. The untraced StandardLink fast path preserves behavior: without a configured tracer neither the outer span nor the three inner runWithSpan wrappers were ever observable, so skipping them changes nothing but cost. The one item worth flagging is the interceptors snapshot in createORPCClient, inlined below.

Reviewed changes

  • Added a module-level WeakMap identity cache to pathToHttpPath so memoized path arrays skip re-encoding.
  • Added single-slot base-URL parse caches to the RPC and OpenAPI link codecs, keyed by the resolved URL value.
  • Memoized OpenAPILinkCodec route resolution (meta, method, structures, merged pathname, dynamic params), shared between encodeInput and decodeResponse, resolving once instead of twice with in-flight dedupe and no caching of failed lazy loads.
  • Added an untraced fast path to StandardLink.call that skips span-name construction and the runWithSpan wrappers when no tracer is configured.
  • Hoisted the client interceptors array out of the per-call closure in createClient.
  • Folded codec depth/params/query scenarios into the existing bench files.

Pullfrog  | Fix all ➔Fix 👍s ➔View workflow run | Using DeepSeek Flash (free via Pullfrog for OSS) | 𝕏

link: ClientLink<InferClientContext<T>>,
{ path = [], ...options }: NoInfer<ORPCClientOptions<T>> = {},
): T {
const clientInterceptors: ORPCClientInterceptor<InferClientContext<T>, unknown, unknown, InferClientError<T>>[] = [

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a silent semantic change to a public API: interceptors and scoped.interceptors are now snapshotted at client creation, so an in-place mutation of the array afterwards (e.g. pushing a later-added interceptor) is no longer observed by subsequent calls. The PR body documents this, but nothing in the code or the option's JSDoc telegraphs it, and the inner ...toArray(options.interceptors) reads previously made the live-read semantics reasonably discoverable. Worth a short comment here noting the snapshot — nothing blocking, the change itself is a good perf win.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ Re-review of the new commit (73fc524 "fix: preserve cache and promise semantics") — no new issues found.

This run re-reviews 73fc524, which sits on top of the previously-reviewed 41e3985 + f2a3e54. It hardens the memoization from the first review, and each fix is genuinely pinned by a test that fails without the change. I verified the affected suites — http.test.ts, openapi-link-codec.test.ts, and link.test.ts — 144 tests pass. The prior minor thread on the client.ts interceptors snapshot (a requested inline note) is unchanged by this commit and stays open below the approval gate; it is non-blocking.

Reviewed changes

  • Changed pathToHttpPath to store [path.slice(), httpPath] in its WeakMap and validate on every hit (length + .every(===)), so reusing a mutated array can no longer return a stale encoded path. The new mutation-invalidation test locks this in.
  • Wrapped the untraced StandardLink.call interceptor invocation in Promise.resolve/Promise.reject so a synchronously-throwing interceptor yields a rejected promise instead of a synchronous throw, matching the traced path and the Promise return type. New test asserts the promise rejects with the thrown error.
  • Deferred OpenAPILinkCodec.computeRoute behind Promise.resolve().then(...) and snapshotted the path, so a synchronously-throwing lazy loader self-cleans from routePromises in finally (never cached) and an in-flight lazy path array mutated mid-load cannot corrupt resolution. Two tests pin the not-cached (loads === 2) and stable-path behaviors.

ℹ️ Nitpicks

  • pathToHttpPath now pays O(depth) for the length check + .every on every cache hit — the previously O(1) fast path this PR optimizes. The trade-off is fair (still far cheaper than encodeURIComponent + join), but since this is a perf PR it's worth re-running the depth benchmarks after this commit to confirm the memoized-array hot path didn't regress materially; trusting the documented stable-array contract is an alternative if it did.
Technical details
# Snapshot-validation cost on pathToHttpPath

## Affected sites
- packages/shared/src/http.ts:12-18 — per-hit length + `.every(===)` scan.

## Required outcome
- Confirm the added O(depth) snapshot check does not materially regress the memoized-array hot path measured in `benches/rpc-link-handler.bench.ts` / `openapi-link-handler.bench.ts`. If it does, prefer trusting the documented stable-array contract (mutation-miss) over validating every hit.

## Open questions for the human
- Is in-place mutation of a shared client path array a real scenario, or is the mutation guard theoretical? If theoretical, the snapshot may not be worth its per-hit cost.

Pullfrog  | Fix it ➔View workflow run | Using DeepSeek Flash (free via Pullfrog for OSS) | 𝕏

@spa5k
spa5k force-pushed the perf/client-codec-memoization branch from 73ff524 to 92a40d2 Compare August 25, 2026 09:41
spa5k added 3 commits August 25, 2026 15:22
- cache pathToHttpPath results by path identity (paths are stable arrays)
- single-slot cache for base URL parsing in RPC and OpenAPI link codecs
- memoize OpenAPILinkCodec route resolution (meta, path template, dynamic
  params) with in-flight dedupe, keyed collision-free; was resolved twice
  per call, once in encodeInput and once in decodeResponse
- untraced fast path in StandardLink.call: build span names and wrap
  encode/send/decode with runWithSpan only when a tracer is configured
- hoist the invariant client interceptors array out of the per-call closure

Includes adversarial tests: dotted-vs-nested route key isolation (the
join('.') cache key collided), lazy contract resolution with failures
not cached, alternating per-call base urls, untraced StandardLink
round-trip, and pathToHttpPath cache consistency.
…sting files

- rpc-link-handler: codec path-encoding depth (10/20, memoized paths) plus
  the fresh-array cache-miss worst case
- openapi-link-handler: codec route-resolution depth (10/20), 1000-procedure
  router, 5 dynamic path params, 50-param query encoding
@spa5k
spa5k force-pushed the perf/client-codec-memoization branch from 92a40d2 to ce45443 Compare August 25, 2026 09:56
* Procedure paths are usually stable arrays reused across calls. Keep a
* snapshot so mutable arrays cannot return a stale encoded path.
*/
const HTTP_PATH_CACHE = new WeakMap<readonly string[], [readonly string[], `/${string}`]>()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not worth to introducing a global WEAKMAP

const baseUrl = await value(this.baseUrl, options, path, input)

const [pathname, search, hash] = parseStandardUrl(baseUrl)
const [pathname, search, hash] = this.parseBaseUrl(baseUrl)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parseStandardUrl is fast enough

private readonly serializer: Exclude<OpenAPILinkCodecOptions<T>['serializer'], undefined>
private readonly customErrorResponseBodyDecoder: OpenAPILinkCodecOptions<T>['customErrorResponseBodyDecoder']
private parsedBaseUrl: [StandardUrl, `/${string}`, `?${string}` | undefined, `#${string}` | undefined] | undefined
private readonly routes = new Map<string, ResolvedOpenAPIRoute>()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TO support this pattern, we shouldn't compute routes: https://orpc.dev/docs/contract/client-factory

@dinwwwh dinwwwh closed this Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants