Skip to content

perf(server): reduce middleware pipeline overhead - #1950

Closed
spa5k wants to merge 4 commits into
middleapi:mainfrom
spa5k:perf/middleware-pipeline
Closed

perf(server): reduce middleware pipeline overhead#1950
spa5k wants to merge 4 commits into
middleapi:mainfrom
spa5k:perf/middleware-pipeline

Conversation

@spa5k

@spa5k spa5k commented Aug 25, 2026

Copy link
Copy Markdown

What this PR does

The middleware pipeline does less work per request. The API does not change. Runtime code changes are limited to packages/server/src/procedure-client.ts.

What changed

  1. The execution plan is computed one time per procedure. A WeakMap holds it. Before, the code computed it again for each request.
  2. The OpenTelemetry config is read one time per call. When no tracer is set, the code calls the middleware, the handler, and the validators directly. This skips the async runWithSpan wrapper. It also skips the span name strings. Span names and attributes do not change when a tracer is set.
  3. The code does not copy the context when the middleware passes no new data. The check sees string keys and symbol keys. The rate limit middleware passes its data through a symbol key. A test covers this case.
  4. Small changes:
    • Primitive outputs skip the instanceof ORPCError check.
    • Non-lazy procedures skip the unlazy promise.
    • The error constructor map and the error reconcile function are cached per procedure.

How much faster

Test setup: Apple M4, Node v25.9.0, Vitest v4.1.11, tracing off. The base is current main.

Test Before After Change
Procedure call, 10 middlewares 2.84µs 1.97µs 1.44x faster
Procedure call, 100 middlewares 26.61µs 15.58µs 1.71x faster
Procedure call, 50 middlewares + 51 input schemas 29.35µs 19.66µs 1.49x faster
Procedure call, validated 0.97µs 0.75µs 1.29x faster
Procedure call, full stack 1.77µs 1.33µs 1.33x faster
OpenAPI e2e, POST with path param and body 6.07µs 5.43µs 1.12x faster

The largest gain is on deep pass-through middleware chains.

Benchmark scenarios

The PR adds scenarios to the existing benchmark files. It adds no new files.

  • procedure-call.bench.ts: middleware counts 10 and 100 (passthrough), 10 and 50 (context-adding), 10 and 50 with stacked input schemas.
  • rpc-link-handler.bench.ts: small payloads. The scenarios cover plain, middleware x3, error, and 404.
  • openapi-link-handler.bench.ts: small payloads with GET and POST on a dynamic path.
  • rpc-serializer.bench.ts: payloads with plain JSON only.

The old scenario names stay the same. CodSpeed history stays comparable.

Behavior notes

  • Spans do not change when tracing is on.
  • When a middleware passes no context, the next level gets the same context object. It does not get a copy. The old code made a copy. No test sees this difference.
  • The OpenTelemetry config is read one time per procedure call. Before, the code read it one time per span.

Tests

  • Focused server tests: 45 passed.
  • Bun 1.4.0 coverage suite: 97 passed and 31 skipped.
  • Node 24 coverage run: 3266 passed and 41 skipped. procedure-client.ts has 100 percent line coverage.
  • The full coverage command then reported the pre-existing Node upload cancellation defect isolated in fix(node): drain rejected upload bodies #1952.
  • New tests cover untraced validation, empty context objects, next(undefined), symbol keys, primitive outputs, and the span name sequence with a mock tracer.
  • eslint and package diagnostics pass with no errors.

A second PR is stacked on this one. It speeds up the client codecs. I will link it here.

spa5k added 3 commits August 25, 2026 13:30
- precompute per-procedure execution plans (schema slices, flags) in a WeakMap
- skip async runWithSpan wrappers and eager span names when tracing is disabled
- skip context spreads when middleware passes an empty context (string- and symbol-keyed aware)
- skip unlazy promise allocation for non-lazy procedures
- cache per-procedure error constructor map and reconcileError closure
- skip the prototype-chain-walking instanceof ORPCError check for primitive outputs
- procedure-call: middleware scaling (10/100 passthrough, 10/50
  context-adding, 10/50 with stacked input schemas)
- rpc-link-handler: tiny-payload fixed overhead scenarios
  (plain, middlewares x3, error thrown, 404)
- openapi-link-handler: tiny dynamic-path GET/POST scenarios
- rpc-serializer: pure-JSON payload scenarios
@spa5k

spa5k commented Aug 25, 2026

Copy link
Copy Markdown
Author

Stacked follow-up: #1951 (client codec memoization). Review it after this one. Its last three commits are the new work. The first four commits belong to this PR.

@pkg-pr-new

pkg-pr-new Bot commented Aug 25, 2026

Copy link
Copy Markdown
More templates

@orpc/ai-sdk

npm i https://pkg.pr.new/@orpc/ai-sdk@1950

@orpc/arktype

npm i https://pkg.pr.new/@orpc/arktype@1950

@orpc/bun

npm i https://pkg.pr.new/@orpc/bun@1950

@orpc/client

npm i https://pkg.pr.new/@orpc/client@1950

@orpc/cloudflare

npm i https://pkg.pr.new/@orpc/cloudflare@1950

@orpc/contract

npm i https://pkg.pr.new/@orpc/contract@1950

@orpc/experimental-effect

npm i https://pkg.pr.new/@orpc/experimental-effect@1950

@orpc/evlog

npm i https://pkg.pr.new/@orpc/evlog@1950

@orpc/hibernation

npm i https://pkg.pr.new/@orpc/hibernation@1950

@orpc/json-schema

npm i https://pkg.pr.new/@orpc/json-schema@1950

@orpc/experimental-msw

npm i https://pkg.pr.new/@orpc/experimental-msw@1950

@orpc/nest

npm i https://pkg.pr.new/@orpc/nest@1950

@orpc/next

npm i https://pkg.pr.new/@orpc/next@1950

@orpc/node

npm i https://pkg.pr.new/@orpc/node@1950

@orpc/openapi

npm i https://pkg.pr.new/@orpc/openapi@1950

@orpc/opentelemetry

npm i https://pkg.pr.new/@orpc/opentelemetry@1950

@orpc/pinia-colada

npm i https://pkg.pr.new/@orpc/pinia-colada@1950

@orpc/pino

npm i https://pkg.pr.new/@orpc/pino@1950

@orpc/publisher

npm i https://pkg.pr.new/@orpc/publisher@1950

@orpc/ratelimit

npm i https://pkg.pr.new/@orpc/ratelimit@1950

@orpc/server

npm i https://pkg.pr.new/@orpc/server@1950

@orpc/shared

npm i https://pkg.pr.new/@orpc/shared@1950

@orpc/swr

npm i https://pkg.pr.new/@orpc/swr@1950

@orpc/tanstack-query

npm i https://pkg.pr.new/@orpc/tanstack-query@1950

@orpc/trpc

npm i https://pkg.pr.new/@orpc/trpc@1950

@orpc/valibot

npm i https://pkg.pr.new/@orpc/valibot@1950

@orpc/zod

npm i https://pkg.pr.new/@orpc/zod@1950

commit: 466cff4

@codecov

codecov Bot commented Aug 25, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.29730% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
packages/server/src/procedure-client.ts 97.29% 3 Missing ⚠️

📢 Thoughts on this report? Let us know!

@codspeed-hq

codspeed-hq Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will improve performance by 12.78%

⚡ 2 improved benchmarks
✅ 28 untouched benchmarks
🆕 14 new benchmarks

Performance Changes

Benchmark BASE HEAD Efficiency
validated 76.9 µs 67.8 µs +13.45%
octet stream 912.2 µs 813.7 µs +12.11%
🆕 get dynamic path param N/A 463.5 µs N/A
🆕 post dynamic path param + body N/A 442.3 µs N/A
🆕 10 middlewares (context-adding) N/A 267.1 µs N/A
🆕 10 middlewares (passthrough) N/A 131.1 µs N/A
🆕 10 middlewares + 11 stacked input schemas N/A 184.9 µs N/A
🆕 100 middlewares (passthrough) N/A 472.4 µs N/A
🆕 50 middlewares (context-adding) N/A 1.6 ms N/A
🆕 50 middlewares + 51 stacked input schemas N/A 541.7 µs N/A
🆕 error thrown N/A 520.2 µs N/A
🆕 middlewares x3 N/A 317.6 µs N/A
🆕 not found (404) N/A 338.9 µs N/A
🆕 plain (no schema, no middleware) N/A 271 µs N/A
🆕 100KB payload (pure JSON) N/A 1.6 ms N/A
🆕 1KB payload (pure JSON) N/A 95.1 µs N/A

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing spa5k:perf/middleware-pipeline (466cff4) with main (2991776)

Open in CodSpeed

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

I reviewed the whole diff and traced the pipeline logic line-by-line. The change is a clean, well-scoped refactor with no functional regressions.

Reviewed changes

  • Per-procedure caching: Two WeakMaps keyed on the procedure — getProcedureCallArtifacts caches the error-constructor map (which allocates a Proxy) and reconcileError; getExecutionPlan precomputes the per-level schema slice boundaries, the stacked-object snapshot, and validation flags once instead of per request.
  • Untraced fast path: getOpenTelemetryConfig()?.tracer is read once per call; when no tracer is configured the middleware/validators/handler run directly, skipping the runWithSpan wrapper. Span names and attribute side effects are unchanged when traced.
  • Conditional context copy: hasEnumerableProperties decides whether to spread-merge or reuse the same context object when a middleware adds no data; symbol keys are handled. Syntactic equivalence to the old always-copy path holds for the merged content.
  • Micro-opts: primitives skip the instanceof ORPCError branch, non-lazy procedures skip the unlazy resolved-promise, and next context extraction reads rest[0]?.context directly.
  • Tests & benches: new tests meaningfully exercise the empty-{}/next(undefined)/symbol-key context cases, the context: undefined result, primitive outputs, and the exact traced span-name sequence. Benches extend existing files. Full test file passes (44/44).

ℹ️ Middleware context now aliases across pass-through frames

This is a documented, deliberate tradeoff, not a bug — but it's slightly subtler than "the next level gets the same object". Previously every next() performed { ...context, ...nextContext }, which gave each middleware frame an isolated copy. On the new path, when a middleware passes no context data the identical object reference flows down, so an in-place mutation of options.context inside a middleware is now visible to ancestor frames that previously received their own copy. The content-equality contract is intact; only in-place mutation aliasing changes. Given use() snapshots schemas into immutable OrderedMiddleware entries and contexts are conventionally read-only, this looks safe — just flagging it as the one caveat reviewers should keep in mind rather than a change to request.

Technical details
# Context object aliasing across pass-through middleware frames

## Affected sites
- packages/server/src/procedure-client.ts:379-389 (the middleware `next` callback: `hasEnumerableProperties` decides merge-copy vs. reuse)
- packages/server/src/procedure-client.ts:406-409 (result context reuse)

## Required outcome
- No change required. Content-merge semantics are preserved; only object identity differs from the previous always-copy behavior.
- Document, if not already, that middlewares must not mutate `options.context` in place and rely on isolation, since passthrough frames now share the same object.

Pullfrog  | View workflow run | Using DeepSeek Flash (free via Pullfrog for OSS) | 𝕏

@dinwwwh

dinwwwh commented Aug 25, 2026

Copy link
Copy Markdown
Member

I'm still not convinced by this approach. It only saves <1µs per request while using a WeakMap, introducing additional complexity, increase bundle size.

@spa5k

spa5k commented Aug 25, 2026

Copy link
Copy Markdown
Author

I don't agree with the bundle size tbf since it runs in the backend not frontend, but for the rest, let me think.

@spa5k

spa5k commented Aug 25, 2026

Copy link
Copy Markdown
Author

but you are still right, the performance improvement isnt that much, want me to close it?

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Since the prior pullfrog review (466cff4), this run reviewed commit 5eead6e:

  • Added a test covering the untraced input/output validation fast path — a transformed .input(...)/.output(...) pipeline is exercised with the no-op tracer disabled, so the test would fail if the fast path skipped transform application.
  • Simplified hasEnumerableProperties by dropping the now-redundant undefined early-return and narrowing its parameter to Context; both call sites (nextContext !== undefined && … and resultContext !== undefined && …) already guard against undefined before invoking.

The narrowed signature is type-safe and behavior-neutral, and the new test meaningfully exercises the branch it claims to cover. The full test file passes (45/45).

Pullfrog  | View workflow run | Using DeepSeek Flash (free via Pullfrog for OSS) | 𝕏

@spa5k spa5k closed this Aug 25, 2026
@spa5k

spa5k commented Aug 25, 2026

Copy link
Copy Markdown
Author

Closing this after considering the review feedback. On normal request paths, the measured savings are about 0.2–0.9µs per request. The larger gains require unusually deep middleware chains. The implementation adds two WeakMap caches, extra branching, more code, and a context object-identity change, so the maintenance and bundle-size cost is not justified by the typical gain.

Thanks for the review. #1951 has been rebased onto main and no longer contains this PR's changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants