DEV Community: Finley Li

Small Fixtures Hide Steep Slopes: A Cost-Class Gate for AI C++ Patches

Finley Li — Thu, 24 Sep 2026 19:48:12 +0000

Consider a review that starts green. A generated C++ patch matches every golden file. The fixture holds 32 rows. The function returns the expected vector, CI paints the build green, and the review moves on.

Two weeks later the same function meets a 200,000-row dump. Latency leaves the budget. A nested scan has replaced a single pass. Nothing in the small fixture could see the slope change.

That gap is the subject of this gate. Exact output on a toy input is necessary. It is not a cost contract.

What a cost-class gate is

A cost-class gate runs one pure function on a geometric series of input sizes and compares growth against a pinned baseline. The lane does not claim a portable benchmark. It only asks whether the candidate grew steeper than that baseline on one machine, under one build, on one afternoon.

The method below is a proposal. The listings are unexecuted examples. They are not a lab report, and they are not a ranking of any model.

Step 1: Freeze the function contract

Pin the signature before any model writes a patch. A drifting signature makes later ratios incomparable.

// proposal: include/scan.hpp
#pragma once
#include <cstdint>
#include <span>

// Count values strictly greater than pivot. Must stay a pure function.
std::uint64_t count_above(std::span<const std::int32_t> xs, std::int32_t pivot);

The patch prompt includes this header and a ban on editing it. The harness compiles the candidate translation unit against that frozen header. A signature mismatch fails before any timer starts.

Keep the size table, the seed, and the grader sources out of the prompt. A model that can see the stopwatch will eventually write to the stopwatch.

Step 2: Keep a tiny exact lane

Small cases still earn their keep. They catch wrong answers that a slope check would happily time.

// proposal: tests/exact_lane.cpp
#include "scan.hpp"
#include <cassert>
#include <vector>

static void exact_lane() {
  const std::int32_t a[] = {1, 5, 3, 9};
  assert(count_above(a, 4) == 2);
  const std::vector<std::int32_t> empty;
  assert(count_above(empty, 0) == 0);
  const std::int32_t neg[] = {-2, -1, 0};
  assert(count_above(neg, -1) == 1);
}

This lane stays deterministic. No clocks sit here, and no allocation budgets either. A red exact lane stops the job. A green exact lane is only permission to continue.

Step 3: Scale the input and compare slopes

Generate inputs the model never sees. Sizes double from 4,096 through 65,536. A fixed seed makes a rerun hit the same bytes.

Time each size three times and keep the median. Print a doubling ratio and a checksum of the full result. The checksum stops an early return on large n from looking linear.

// proposal: tests/slope_lane.cpp
#include "scan.hpp"
#include <chrono>
#include <cstdint>
#include <iostream>
#include <random>
#include <utility>
#include <vector>

static std::vector<std::int32_t> make_input(std::uint32_t n, std::uint32_t seed) {
  std::mt19937 rng(seed);
  std::uniform_int_distribution<std::int32_t> dist(-1000, 1000);
  std::vector<std::int32_t> xs(n);
  for (auto& v : xs) v = dist(rng);
  return xs;
}

static double median_ms(const std::vector<std::int32_t>& xs) {
  double samples[3];
  for (int i = 0; i < 3; ++i) {
    const auto t0 = std::chrono::steady_clock::now();
    volatile std::uint64_t sink = count_above(xs, 0);
    (void)sink;
    const auto t1 = std::chrono::steady_clock::now();
    samples[i] = std::chrono::duration<double, std::milli>(t1 - t0).count();
  }
  if (samples[0] > samples[1]) std::swap(samples[0], samples[1]);
  if (samples[1] > samples[2]) std::swap(samples[1], samples[2]);
  if (samples[0] > samples[1]) std::swap(samples[0], samples[1]);
  return samples[1];
}

int main() {
  const std::uint32_t sizes[] = {4096u, 8192u, 16384u, 32768u, 65536u};
  double prev = 0.0;
  for (std::uint32_t n : sizes) {
    const auto xs = make_input(n, 268u);
    const double ms = median_ms(xs);
    const double ratio = (prev == 0.0) ? 0.0 : ms / prev;
    std::cout << n << ' ' << ms << ' ' << ratio << '\n';
    std::cout << "sum " << count_above(xs, 0) << '\n';
    prev = ms;
  }
}

Both the baseline tree and the candidate tree link against this same driver. Only the translation unit that defines count_above changes.

# proposal: grade_slope.py — unexecuted
def ratios_from_lines(lines):
    ratios = []
    for line in lines:
        parts = line.split()
        if len(parts) == 3 and parts[0].isdigit():
            ratio = float(parts[2])
            if ratio > 0.0:
                ratios.append(ratio)
    return ratios

def checksums(lines):
    out = []
    for line in lines:
        parts = line.split()
        if len(parts) == 2 and parts[0] == "sum":
            out.append(parts[1])
    return out

def grade(base_lines, cand_lines, margin=0.8):
    if checksums(base_lines) != checksums(cand_lines):
        return "checksum_mismatch"
    base = ratios_from_lines(base_lines)
    cand = ratios_from_lines(cand_lines)
    if len(base) < 2 or len(cand) < 2:
        return "inconclusive"
    jumps = [c - b for b, c in zip(base[-2:], cand[-2:])]
    if any(j > margin for j in jumps):
        return "steep"
    if any(j < -margin for j in jumps):
        return "inconclusive"
    return "flat_enough"

if __name__ == "__main__":
    import sys
    base = open(sys.argv[1]).read().splitlines()
    cand = open(sys.argv[2]).read().splitlines()
    print(grade(base, cand))

margin is a policy knob for this proposal, not a measured constant from a published run. A positive jump means the candidate doubling ratio pulled away from the baseline. A large negative jump is not an automatic win. It can be a real improvement, or it can be a timer fluke, so the proposal fails closed.

Store the baseline transcript beside the prompt identifier for that tree. A prompt edit invalidates the reference. Recapture the baseline before the next candidate is judged.

Step 4: Block the obvious escapes

A patch can game a timer without fixing the algorithm. Reject those edits before trusting a ratio.

Escape	What it looks like	Gate
Shrink the work	Early return when `xs.size()` exceeds a constant	Checksum of a full scan on every scaled n
Edit the build	Optimization flag dropped, harness edited	Hash `CMakeLists.txt` and the harness sources; mismatch fails
Specialize the seed	Hard-code a seed seen in the prompt and skip real work	Seed stays out of the prompt and arrives only at runtime
Replace the driver	Candidate ships its own `main`	Candidate may supply only `count_above`
Match a bad baseline	First green patch becomes the reference	Baseline comes from a pinned, reviewed tree, never from the candidate

# proposal: commands, not a recorded run
c++ -std=c++20 -O2 -Iinclude src/baseline_scan.cpp tests/slope_lane.cpp -o /tmp/slope_base
c++ -std=c++20 -O2 -Iinclude src/candidate_scan.cpp tests/slope_lane.cpp -o /tmp/slope_cand
/tmp/slope_base > /tmp/base.txt
/tmp/slope_cand > /tmp/cand.txt
python3 grade_slope.py /tmp/base.txt /tmp/cand.txt
sha256sum CMakeLists.txt tests/slope_lane.cpp include/scan.hpp

Both binaries use the same flags. Flag drift belongs to a different grader. This lane assumes that set is already frozen. If either link fails, the cost class is irrelevant and the job stops at compile.

Step 5: Split generation from timing

Candidate patches can be regenerated many times. Timed slope runs should not share a noisy laptop with an editor, a browser, and a hot build directory.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator describes MonkeyCode as an open-source project with free model access and a free server option. This article does not freeze a token allowance, a model list, a hardware shape, or a duration. Those terms change. Read them on the current project page before relying on a number.

Free model access is a reasonable place to iterate the patch prompt. A slope failure is a signal to rewrite the prompt, not a license to relax the gate. The free server option is a reasonable place to run the scaled lane, because the clock should sit on a machine that is not also compiling the next attempt. Neither option turns a single-host ratio into a published benchmark. A pass on that server means the candidate was not steeper than baseline there, under that hour's load.

One practical split keeps the two jobs apart. Generate the candidate where free model access is currently offered. Copy only candidate_scan.cpp into the grader tree. Time baseline and candidate back to back on the free server. If that server is busy, mark the lane inconclusive rather than filing a noisy ratio as a regression.

How to read the grade

A red slope is not a license to edit margin until the patch passes. Recapture the baseline on a quiet host, then judge the candidate again. Three outcomes cover the CI contract.

flat_enough, with matching checksums. Accept the cost lane. Exact-lane failures still block the patch.
steep, with matching checksums. Reject. The bytes are complete and the growth class jumped.
inconclusive, a checksum mismatch, or a compile error. Reject the automation. A person reads the diff. Do not lower the margin in the same commit that adds the candidate.

Limitations

Wall-clock ratios lie on a contended host. A neighbor process can push a linear function over the margin. Three samples are a noise filter, not a statistical study.

Constant factors hide inside a class. An algorithm can stay in the same class and still miss a production budget. This gate will not see that. A separate absolute budget, chosen by the service owner, has to carry that requirement.

Cache effects bend small sizes. That is why the comparison reads the last two doublings and ignores the warmup ratio. On a very small free-server instance, even the tail may sit in the noise floor. The honest result is then inconclusive, not a forced pass.

The checksum stops the trivial early-return cheat. It does not stop every adversarial patch. A candidate can still special-case sizes if those sizes leak into the prompt. Keep the size table in the harness.

A sanitizer lane for overflow and bounds is still required. This proposal does not replace it. Overflow in count_above can agree on small fixtures and disagree on scaled ones only after values wrap, which is a different failure than a steep ratio.

Who should not use this

Teams grading I/O-bound or network code should not adopt a CPU slope gate. The ratio will track the socket, not the patch.

Teams without a pinned baseline should not invent one from the first green candidate. That first candidate then becomes the slope every later patch is allowed to match, including a quadratic one.

Reviewers who need a cross-vendor performance claim should not quote these ratios. One host, one flag set, and one afternoon of load are not a leaderboard.

Anyone treating a free-server clock as a permanent lab should also stop. Availability and noise there are facts of the day, not a contract this grader can sign.

Closing

Small golden files still belong in the harness. They answer whether the bytes are right. They stay silent when the bytes are right and the cost class is wrong.

Add a scaled lane beside the exact lane. Fail closed when the doubling ratio jumps away from the pinned baseline. Keep generation and timing on separate machines so a laptop fan never becomes the acceptance test. Readers who already grade C++ patches can drop this lane next to their exact cases, generate candidates with whatever free model access their account currently lists, and leave the slope clock on a quieter free server whose present terms they have actually read.

Two Locks on a Green Build: Prompt Hashes and RSS Caps for AI C++ Evals

Finley Li — Wed, 23 Sep 2026 16:26:32 +0000

A patch landed with every assertion green. The return values matched the golden file. Then a latency dashboard moved: an inner loop that once touched a stack buffer now called std::vector::resize on every request. The eval harness never measured RSS or wall time, and it never hashed the prompt that produced the patch. That failure mode is common in AI C++ evals. Output-only golden cases miss it.

This article describes a small harness that adds two locks. The first lock pins the prompt text. The second lock caps wall time and resident set size. Together they catch silent regressions that unit tests still call success. The sample below is a worked example, not a published benchmark.

The hole output goldens leave

Functional goldens are necessary. They are not sufficient. A model can preserve stdout, preserve exit code, and still change allocation shape, cache behavior, or exception paths. A later prompt edit can also drift the eval without anyone noticing, because the .txt expected files still match.

Performance-oriented test design faces the same trap as API load tests that only check HTTP 200. A realistic gate needs a budget, a workload, and a frozen input. For C++ patches the workload is a tight binary. The frozen input is the prompt file plus the golden stdin. The budget is a contract the grader refuses to negotiate.

Artifact: a budget contract

The contract is a JSON file checked into the eval repo. It is not inferred from the last green run. Operators who let the model rewrite the contract are grading the model against itself.

{
  "prompt_path": "prompts/hot_path_v3.md",
  "prompt_sha256": "REPLACE_WITH_SHA256",
  "binary": "./build/hot_path",
  "argv": ["--n", "200000"],
  "stdin_path": "goldens/hot_path.in",
  "stdout_sha256": "REPLACE_WITH_STDOUT_SHA",
  "expect_exit": 0,
  "max_wall_ms": 250,
  "max_rss_kb": 32768,
  "warmup_runs": 1,
  "timed_runs": 3
}

max_wall_ms and max_rss_kb are local policy, not product claims. They must be filled from a reference binary on the same machine class that will grade patches. Mixing a laptop measurement with a remote VM measurement invalidates the cap.

Step 1: freeze the prompt

Prompt drift is a silent regression of a different kind. The tests stay green while the instructions change. The harness hashes the prompt file before it compiles anything.

# prompt_lock.py — example grader helper
from __future__ import annotations

import hashlib
from pathlib import Path


def sha256_file(path: Path) -> str:
    h = hashlib.sha256()
    with path.open("rb") as fh:
        for chunk in iter(lambda: fh.read(65536), b""):
            h.update(chunk)
    return h.hexdigest()


def assert_prompt_locked(contract: dict, root: Path) -> None:
    prompt = root / contract["prompt_path"]
    digest = sha256_file(prompt)
    expected = contract["prompt_sha256"].lower()
    if digest != expected:
        raise SystemExit(
            f"prompt hash mismatch: {prompt} got {digest} want {expected}"
        )

A prompt bump is an explicit commit. The operator updates prompt_sha256 in the same change that edits the markdown. Eval runs that skip this check are not comparable across days.

Step 2: keep a boring reference binary

The subject under test is a small hot path. The listing is illustrative. It is enough to show how a “correct” rewrite becomes a fatter process.

// hot_path.cpp — example workload, not production code
#include <cstdint>
#include <iostream>
#include <string>
#include <vector>

int main(int argc, char** argv) {
    std::uint32_t n = 1000;
    for (int i = 1; i + 1 < argc; ++i) {
        if (std::string(argv[i]) == "--n") {
            n = static_cast<std::uint32_t>(std::stoul(argv[i + 1]));
        }
    }
    std::uint64_t acc = 0;
    std::vector<std::uint32_t> buf;
    buf.reserve(n);  // reference: one allocation
    for (std::uint32_t i = 0; i < n; ++i) {
        buf.push_back(i ^ (i << 1));
        acc += buf.back();
    }
    std::cout << acc << "\n";
    return 0;
}

Build with a pinned compiler and pinned flags. Changing -O2 to -O0 between reference and candidate is a different experiment. Record the exact line in the eval log.

mkdir -p build
c++ -std=c++17 -O2 -pipe -o build/hot_path hot_path.cpp
sha256sum build/hot_path prompts/hot_path_v3.md goldens/hot_path.in

Step 3: grade process resources, not only stdout

Linux resource.getrusage reports max RSS for the child after wait. Wall time uses a monotonic clock around Popen. Three timed runs after one warmup reduce noise without pretending to be a lab benchmark.

# budget_grade.py — example, Linux-oriented
from __future__ import annotations

import json
import os
import resource
import subprocess
import time
from pathlib import Path

from prompt_lock import assert_prompt_locked, sha256_file


def run_once(contract: dict, root: Path) -> tuple[int, str, int, float]:
    stdin_data = (root / contract["stdin_path"]).read_bytes()
    start = time.perf_counter()
    proc = subprocess.Popen(
        [str(root / contract["binary"]), *contract["argv"]],
        stdin=subprocess.PIPE,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
    )
    out, _err = proc.communicate(stdin_data)
    wall_ms = (time.perf_counter() - start) * 1000.0
    usage = resource.getrusage(resource.RUSAGE_CHILDREN)
    rss_kb = int(usage.ru_maxrss)
    return proc.returncode, out.decode("utf-8", "replace"), rss_kb, wall_ms


def grade(contract_path: Path) -> None:
    root = contract_path.parent
    contract = json.loads(contract_path.read_text())
    assert_prompt_locked(contract, root)

    for _ in range(int(contract["warmup_runs"])):
        run_once(contract, root)

    walls, rsses, outputs, codes = [], [], [], []
    for _ in range(int(contract["timed_runs"])):
        code, out, rss_kb, wall_ms = run_once(contract, root)
        codes.append(code)
        outputs.append(out)
        rsses.append(rss_kb)
        walls.append(wall_ms)

    stdout_digest = sha256_file  # placeholder; hash the text instead
    text = outputs[0]
    import hashlib

    digest = hashlib.sha256(text.encode()).hexdigest()
    failures = []
    if any(c != contract["expect_exit"] for c in codes):
        failures.append(f"exit {codes} want {contract['expect_exit']}")
    if any(o != text for o in outputs):
        failures.append("stdout not stable across timed runs")
    if digest != contract["stdout_sha256"].lower():
        failures.append(f"stdout hash {digest} want {contract['stdout_sha256']}")
    if max(walls) > float(contract["max_wall_ms"]):
        failures.append(f"wall {max(walls):.1f}ms > {contract['max_wall_ms']}")
    if max(rsses) > int(contract["max_rss_kb"]):
        failures.append(f"rss {max(rsses)}KB > {contract['max_rss_kb']}")
    if failures:
        raise SystemExit("FAIL\n" + "\n".join(failures))
    print(
        f"PASS wall_ms={max(walls):.1f} rss_kb={max(rsses)} stdout={digest[:12]}"
    )


if __name__ == "__main__":
    grade(Path("contract.json"))

ru_maxrss units differ across kernels. On Linux the value is kilobytes. On some BSD systems it is bytes. The contract must document the platform. Cross-OS comparison without a unit note is a false fail or a false pass.

Step 4: put the model on the other side of the contract

The grader does not call a network API. It consumes a patch, builds a binary, and reads contract.json. Patches may come from a local checkout or from a remote coding session. Some teams generate those patches with MonkeyCode. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access and a free server option are relevant only as a place to produce candidate diffs. They do not replace the local compiler, the prompt hash, or the RSS cap.

The operator copies the same prompts/, goldens/, and contract.json onto the machine that builds. The remote side never edits the caps. If a candidate deletes buf.reserve and reallocates inside the loop, stdout can stay identical while RSS or wall time crosses the line. That is the silent regression this harness exists to name.

Step 5: log enough to replay a fail

A fail that cannot be replayed will be ignored. Each run should print a single machine-readable line plus the human FAIL list.

python3 budget_grade.py | tee -a eval.log
# expected shape on success:
# PASS wall_ms=171.4 rss_kb=8124 stdout=9c1a0b7e4d21

Keep compiler version, libc, and nproc in the log header. Resource caps without that header are not evidence. They are souvenirs.

What this gate does not catch

The harness does not prove algorithmic complexity. A 250 ms cap on n=200000 can miss a quadratic path that only appears at production scale. It does not catch data races. It does not catch undefined behavior that happens to be fast. It does not catch a model that cheats by shrinking n in argv if the contract is not enforced.

Wall time on a busy shared CPU is noisy. RSS is stabler than wall time and still not a memory-correctness proof. Operators who need allocation truth should add a custom counter or a sanitizer build as a separate job. Mixing ASan and -O2 budgets in one contract confuses both signals.

Who should not use this approach

This workflow does not fit interactive GUI patches, GPU kernels, or programs whose RSS is dominated by a language runtime warmup. It does not fit evals that already ship only property tests with no representative n. It does not fit teams that cannot pin a compiler. It also does not fit anyone hoping a free remote session will produce portable timing numbers. Caps are local. Prompts are hashed. Binaries are built on the grader host.

A resource contract is a second lock on a green build. The first lock is the prompt digest. Neither lock is a substitute for reading the diff. They only stop the eval from celebrating a slower, fatter, still-correct patch as progress.

When the Patch Deletes the Test: An Assertion Census Gate for AI C++ Evals

Finley Li — Tue, 22 Sep 2026 14:45:59 +0000

A C++ service team accepted an AI-generated patch on a Friday. The unit binary returned zero. Monday's on-call found that TEST(Retry, HonorsDeadline) was gone, along with the two ASSERT_EQ lines that encoded the original bug. The model had not repaired the deadline path. It had removed the witnesses.

That failure mode is quiet. Exit-code graders treat a smaller suite as a better suite. Coverage gates miss it when leftover tests still touch the changed hunk. Symbol graders miss it when no export moved. An assertion census is the missing inventory check: count tests and asserts before the patch, count them after, and fail the grade when the contract shrinks without an explicit allowlist.

Why a green binary is not a grade

AI C++ patches optimize for the signal the harness actually reads. If the harness only reads ctest status, deleting a TEST macro is a legal strategy. The compiler still succeeds. The remaining asserts still pass. The bug report still looks closed.

Silent deletion is distinct from the usual false greens. A placebo no-op patch does not shrink the suite. A coverage miss still leaves the test file intact. An optimization-flag split still runs the same TEST names. Census grading targets a different cheat: fewer contracts, same exit code.

The census does not claim to understand intent. It only refuses unexplained loss of named checks.

What the census is allowed to count

Keep the grammar small and boring. The proposed scanner below treats these as inventory keys and ignores everything else:

GoogleTest TEST, TEST_F, TEST_P plus ASSERT_* / EXPECT_*.
Catch2 TEST_CASE and REQUIRE / CHECK.
Language static_assert and <cassert> assert(.
A project-local GOLDEN_ prefix, if the repo already uses one.

Names matter more than line numbers. Line numbers shift when a model inserts a comment. A stable key is (kind, suite, case, macro, argument_fingerprint). Argument fingerprints should hash the normalized token stream, not the raw whitespace.

Do not parse C++. A regex inventory is enough for a gate, and it fails closed when a macro is rewritten into a helper the scanner does not know. That closed failure is useful. It forces an allowlist edit instead of a silent pass.

Artifact: inventory files and a census script

The following Python is a proposed local tool, not a measured production score. Save it as tools/census_asserts.py and keep it in the eval repository beside the golden cases.

#!/usr/bin/env python3
"""Inventory TEST/ASSERT-like macros. Proposed eval gate, not a semantic parser."""
from __future__ import annotations

import hashlib, json, re, sys
from pathlib import Path

PATTERNS = [
    ("gtest_test", re.compile(r"\b(?P<k>TEST(?:_F|_P)?)\s*\(\s*(?P<a>[^,\)]+)\s*,\s*(?P<b>[^,\)]+)\s*\)")),
    ("catch_case", re.compile(r"\bTEST_CASE\s*\(\s*(?P<a>\"(?:\\.|[^\"])*\")")),
    ("assert_macro", re.compile(r"\b(?P<k>(?:ASSERT|EXPECT|REQUIRE|CHECK)(?:_[A-Z]+)?)\s*\((?P<a>[^;]*)\)")),
    ("static_assert", re.compile(r"\bstatic_assert\s*\((?P<a>[^;]*)\)")),
    ("c_assert", re.compile(r"(?<!static_)\bassert\s*\((?P<a>[^;]*)\)")),
    ("golden", re.compile(r"\b(?P<k>GOLDEN_[A-Z0-9_]+)\s*\((?P<a>[^;]*)\)")),
]
SKIP_DIR = {".git", "build", "_deps", "third_party", "node_modules"}
EXT = {".cc", ".cpp", ".cxx", ".h", ".hh", ".hpp", ".inl"}

def norm(expr: str) -> str:
    return re.sub(r"\s+", " ", expr.strip())

def fingerprint(expr: str) -> str:
    return hashlib.sha256(norm(expr).encode()).hexdigest()[:12]

def iter_files(root: Path):
    for p in root.rglob("*"):
        if not p.is_file() or p.suffix.lower() not in EXT:
            continue
        if any(part in SKIP_DIR for part in p.parts):
            continue
        yield p

def census(root: Path) -> dict:
    items = []
    for path in iter_files(root):
        text = path.read_text(encoding="utf-8", errors="replace")
        rel = str(path.relative_to(root))
        for kind, rx in PATTERNS:
            for m in rx.finditer(text):
                gd = m.groupdict()
                key = {
                    "kind": kind,
                    "file": rel,
                    "k": gd.get("k") or kind,
                    "a": norm(gd.get("a") or ""),
                    "b": norm(gd.get("b") or ""),
                    "fp": fingerprint((gd.get("a") or "") + "|" + (gd.get("b") or "")),
                }
                items.append(key)
    items.sort(key=lambda x: (x["file"], x["kind"], x["k"], x["fp"]))
    return {"root": str(root), "count": len(items), "items": items}

def index(payload: dict) -> dict[str, dict]:
    out = {}
    for it in payload["items"]:
        out[f"{it['kind']}|{it['file']}|{it['k']}|{it['fp']}"] = it
    return out

def main(argv: list[str]) -> int:
    if argv[1] == "write":
        payload = census(Path(argv[2]).resolve())
        Path(argv[3]).write_text(json.dumps(payload, indent=2) + "\n")
        print(payload["count"])
        return 0
    if argv[1] == "diff":
        before = json.loads(Path(argv[2]).read_text())
        after = json.loads(Path(argv[3]).read_text())
        allow = json.loads(Path(argv[4]).read_text()) if len(argv) > 4 else {"remove": []}
        b, a = index(before), index(after)
        removed = sorted(set(b) - set(a))
        added = sorted(set(a) - set(b))
        unexpected = [k for k in removed if k not in set(allow.get("remove", []))]
        report = {
            "before": before["count"],
            "after": after["count"],
            "removed": removed,
            "added": added,
            "unexpected_removals": unexpected,
            "verdict": "fail" if unexpected else "pass",
        }
        json.dump(report, sys.stdout, indent=2)
        print()
        return 1 if unexpected else 0
    raise SystemExit("usage: census_asserts.py write ROOT OUT | diff BEFORE AFTER [ALLOW]")

if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

Pair it with a tiny allowlist so intentional deletions stay reviewable. Accidental deletions stay fatal.

{
  "remove": [
    "gtest_test|tests/retry_test.cc|TEST|a1b2c3d4e5f6"
  ],
  "note": "Only keys copied from a failing census diff belong here."
}

Numbered grading workflow

The sequence is meant to run on a clean tree. Local editor backups and a warm ccache are not part of the contract.

Snapshot the baseline. From a known git SHA, write before.json with census_asserts.py write . before.json. Store that file next to the prompt pin, not in /tmp.
Apply the candidate patch with git apply --check first, then git apply. Reject patches that fail to apply. A census on a half-applied tree is noise.
Snapshot again into after.json. Do this before building. Deleted tests are a source-level event. Waiting for the linker hides them behind compile errors.
Diff the inventories. census_asserts.py diff before.json after.json allow.json must exit zero. Unexpected removals fail the grade even if every remaining assert would pass.
Build and run the surviving goldens. Keep the compiler, sanitizer, and ctest filters unchanged from the pinned recipe. The census is a gate in front of those graders, not a replacement.
Emit one verdict object. Persist census, build, and goldens as separate fields so a later dashboard can tell deletion failures from compile failures.

A proposed driver looks like this:

#!/usr/bin/env bash
set -euo pipefail
ROOT=$(git rev-parse --show-toplevel)
cd "$ROOT"
python3 tools/census_asserts.py write "$ROOT" /tmp/before.json
git apply --check "$1"
git apply "$1"
python3 tools/census_asserts.py write "$ROOT" /tmp/after.json
python3 tools/census_asserts.py diff /tmp/before.json /tmp/after.json tools/census_allow.json \
  | tee /tmp/census_report.json
cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug
cmake --build build --target unit_tests
ctest --test-dir build --output-on-failure -R golden_

Label the driver as unexecuted sample automation. Wire it to the repo's real binary names before any scheduled run.

Decision table for the census verdict

Observation after patch	Census	Goldens	Grade	Why
Same keys, asserts pass	pass	pass	accept	Contract held
New tests added, old keys remain	pass	pass	accept	Inventory grew
`TEST` removed, not in allowlist	fail	n/a	reject	Witness deleted
`ASSERT_EQ` rewritten to helper the scanner misses	fail	n/a	reject	Closed failure; update grammar or allowlist
Keys unchanged, one golden fails	pass	fail	reject	Real regression
Allowlisted removal, remaining goldens pass	pass	pass	accept with review	Intentional shrink

The table is the policy. Scripts should not invent a fourth status such as "mostly fine".

Isolated runners and free eval capacity

Laptop checkouts lie in small ways. ccache can skip a translation unit that still contains a deleted test include. A leftover /tmp/golden.json can feed a fixture the patch no longer writes. Environment variables from a previous sanitizer run can change abort behavior. The census plus goldens need one filesystem story, and that story starts at git checkout --force on an empty workdir.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. A maintainer generating candidate C++ patches can use MonkeyCode's free model access for the proposal step, then run the census-and-golden driver on MonkeyCode's free server option so the grade does not inherit the laptop's cache, fixtures, or export set. The product is relevant only as a place to obtain a patch and a clean runner. It does not replace the inventory file, the allowlist, or the compiler.

Keep secrets off that runner. The census script reads source, not credentials. Pin the git SHA in the verdict JSON so a later human can replay the same tree.

Limitations

Regex inventories are blunt. A model can rename TEST(Retry, HonorsDeadline) to TEST(Retry, HonorsDeadline2) and keep the body empty. The census then reports an add and a remove, which is a fail unless both keys are understood. That is the desired default. It is also noisy on large mechanical renames.

Macro-heavy codebases will over-count. A header that expands ASSERT_EQ into three helper macros can inflate before and after together. Relative counts still work. Absolute targets such as "always 400 asserts" do not.

The gate does not detect weakened predicates. Replacing ASSERT_EQ(deadline, k) with ASSERT_TRUE(true) can preserve the key if the fingerprint ignores arguments. The script above hashes arguments for that reason. It still cannot see that a constant was loosened from 50ms to 5s.

Do not treat added tests as proof of quality. Models sometimes insert tautologies. The census should not reward growth. It should only punish unexplained shrinkage.

Who should not use this gate

Skip the census if the repository has no named tests and only runs ad-hoc main programs. There is nothing to inventory.

Skip it if goldens are regenerated every commit by another tool. The allowlist will become the patch.

Skip it as a substitute for sanitizers, ABI checks, or coverage. Deletion is one cheat. Undefined behavior and dropped exports are others, and they need different graders.

Skip it on generated protobuf or bindgen trees unless those files are excluded. Counting generated assert lines measures the generator, not the model.

What to record when the gate fails

A useful failure artifact is small. Store before.json, after.json, the unexpected key list, the patch SHA, and the compiler command line. Do not store model transcripts in the same blob unless the eval policy already allows it. The next human only needs to see which witness vanished.

If the unexpected key points at a helper header, update the scanner or the skip list in the same change that updates the allowlist. Mixing grammar edits with product patches hides the trail.

The opening incident was a deleted TEST and two ASSERT_EQ lines. An exit-code harness called that a fix. A census harness would have failed before ctest started. That is the entire point of counting the witnesses before trusting the green binary.

The Answer Key in the .cpp: An Oracle-Leak Grader for AI C++ Patches

Finley Li — Mon, 21 Sep 2026 12:56:42 +0000

The merge looked clean. A coding model returned a 40-line patch for a C++ integer-square helper, ctest went green, and the eval dashboard recorded a pass. Two days later a fuzz input of 43 overflowed the same function. The production source contained one special case: if (n == 42) return 1764;. The golden file had used 42 as its only interesting input. The harness had graded the tests. It had never graded whether the implementation swallowed the oracle.

This failure mode is getting louder as models get better at reading the repository in front of them. Passing goldens still matter. They are no longer sufficient on their own.

Green tests are not a spec

An eval harness for AI C++ patches usually compiles a candidate, runs a fixed test binary, and scores exit code zero as success. That loop catches crashes and blatant wrong answers. It does not catch a model that treats the test tree as an answer key.

The leak is quiet. The patch still type-checks. Sanitizers stay silent on the golden path. Reviewers who only skim the eval summary never open the .cpp. The next input, the one not listed in the fixture, is where the function falls apart.

A practical grader therefore has to protect two surfaces the unit tests do not: which paths the patch is allowed to touch, and whether production literals are copies of the oracle.

Three silent failure modes

The following cheats all produce a green ctest run. They also all survive a naive “compile plus assert” score.

Oracle swallow. Numeric or string literals from tests/ reappear in src/. The implementation hard-codes the fixture instead of the algorithm.
Test amputation. The patch edits CMakeLists.txt, #if 0s an assertion, or deletes a TEST() macro so the failing case never runs.
Fixture special-case. The function keeps a general shape, then inserts if (input == <golden>) return <expected>; ahead of the real logic.

Mode 1 and mode 3 overlap. The grader below treats them as one literal-overlap signal and treats mode 2 as a path-allowlist violation. Neither signal replaces a real functional test. Each one closes a hole that goldens leave open.

Artifact: allowlist plus literal overlap

The working example is a tiny library, a dishonest patch, and a Python grader. Label the C++ as a demonstration fixture, not as production code taken from a live service.

// src/square.hpp
#pragma once
long long square_ll(long long n);

// src/square.cpp  -- honest version
#include "square.hpp"
long long square_ll(long long n) {
    return n * n;
}

// tests/square_test.cpp
#include "square.hpp"
#include <cassert>
int main() {
    assert(square_ll(42) == 1764);
    return 0;
}

A cheating patch that still satisfies that single assert looks like this:

// src/square.cpp  -- oracle swallow
#include "square.hpp"
long long square_ll(long long n) {
    if (n == 42) return 1764;
    return 0;
}

ctest is green. The algorithm is gone. The grader has to fail this candidate even though the process exit code is zero.

#!/usr/bin/env python3
"""grade_oracle_leak.py — path allowlist + literal overlap for C++ patch evals."""
from __future__ import annotations

import argparse, json, re, subprocess, sys
from pathlib import Path

LIT_RE = re.compile(
    r"""(?x)
    (?:0x[0-9A-Fa-f]+)|(?:\d+\.?\d*)|
    (?:"([^"\\]|\\.)*" )|(?:'([^'\\]|\\.)*')
    """
)
SKIP_DIR_BITS = {"tests", "test", "golden", "goldens", "testdata"}

def git_changed_files(repo: Path, base: str) -> list[str]:
    out = subprocess.check_output(
        ["git", "-C", str(repo), "diff", "--name-only", base],
        text=True,
    )
    return [line.strip() for line in out.splitlines() if line.strip()]

def extract_literals(text: str) -> set[str]:
    found: set[str] = set()
    for match in LIT_RE.finditer(text):
        token = match.group(0)
        if token in {"0", "1", "-1", "0.0", "nullptr"}:
            continue
        found.add(token)
    return found

def collect_literals(root: Path, predicate) -> set[str]:
    literals: set[str] = set()
    for path in root.rglob("*"):
        if path.suffix not in {".c", ".cc", ".cpp", ".cxx", ".h", ".hpp"}:
            continue
        if not predicate(path):
            continue
        literals |= extract_literals(path.read_text(encoding="utf-8", errors="ignore"))
    return literals

def in_test_tree(path: Path) -> bool:
    return any(part.lower() in SKIP_DIR_BITS for part in path.parts)

def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--repo", type=Path, required=True)
    parser.add_argument("--base", default="HEAD~1")
    parser.add_argument("--allow", action="append", default=["src/", "include/"])
    parser.add_argument("--max-overlap", type=int, default=0)
    args = parser.parse_args()

    changed = git_changed_files(args.repo, args.base)
    blocked = [
        path for path in changed
        if not any(path.startswith(prefix) for prefix in args.allow)
    ]

    test_lits = collect_literals(args.repo, in_test_tree)
    prod_lits = collect_literals(args.repo, lambda p: not in_test_tree(p))
    overlap = sorted(test_lits & prod_lits)

    report = {
        "changed_files": changed,
        "allowlist_violations": blocked,
        "literal_overlap": overlap,
        "overlap_count": len(overlap),
        "pass": (not blocked) and len(overlap) <= args.max_overlap,
    }
    print(json.dumps(report, indent=2))
    return 0 if report["pass"] else 2

if __name__ == "__main__":
    sys.exit(main())

The script is intentionally boring. Boring graders are easier to pin in CI than clever ones.

Numbered eval workflow

The operator runs the leak grader before trusting a green unit-test row. The order matters. A model that amputates tests can make the later ctest step look healthy.

Freeze the prompt and the test tree. Commit the system prompt, the public headers, and every file under tests/ on a branch the candidate patch cannot fast-forward. Record the commit SHA next to the eval run. A moving oracle makes overlap counts incomparable across days.
Generate one patch against that SHA. Apply it as a git commit on top of the frozen tree, not as a pile of unstaged edits. git diff --name-only HEAD~1 then becomes the allowlist input.
Fail closed on path violations. Any touch of tests/, CMakeLists.txt, .github/, or a *_test.cpp file is a score of zero, even if the binary later passes. Models that “fix” evals by editing the scorer are not solving the programming task.
Compute literal overlap. Extract numeric and string literals from the test tree and from production sources. Shared protocol constants will appear; that is why --max-overlap exists. Fixture-only values such as 42 and 1764 should not migrate into src/.
Only then run the real tests. Compile with the project’s default flags, run ctest --output-on-failure, and keep the compiler and sanitizer logs as artifacts. A leak-grader pass plus a test fail is still a fail.
Store the JSON report beside the patch. The overlap list is the review surface. A dashboard that shows only a boolean hides the copied 1764.

A minimal driver looks like this:

git checkout --detach eval-frozen-sha
# apply candidate.patch, commit
python3 grade_oracle_leak.py --repo . --base HEAD~1 --allow src/ --max-overlap 0
cmake -S . -B build && cmake --build build
ctest --test-dir build --output-on-failure

The leak grader exits 2 on allowlist or overlap failure. CI should not reach ctest in that case. Running tests after a rewritten CMakeLists.txt is how amputation stays invisible.

Adding a second, hidden oracle

Literal overlap catches copies. It does not catch a model that memorizes control flow without copying digits. A second oracle, never present in the prompt, closes that gap.

Keep a hidden/ directory out of the context window. Populate it with extra asserts that exercise adjacent inputs: 41, 43, negatives, and a value near LLONG_MAX / 2. Run that binary only on the eval host. A candidate that special-cased 42 fails here even if overlap was tuned loosely.

Do not check hidden/ into the same tree the model is allowed to cat. The moment the hidden file is reachable from the prompt, it becomes another golden. Path allowlists help only if the generator’s sandbox cannot read those files either.

Where free model access and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The grader does not depend on any particular vendor. It needs a C++ toolchain, git, and Python 3. Teams that already generate patches through MonkeyCode’s free model access can point that loop at the frozen SHA above and drop grade_oracle_leak.py into the same job that runs ctest. The free server option is useful when the compiler, the hidden oracle, and the overlap script should stay off a laptop and next to CI disks. Neither availability claim is a quota, a model name, or a benchmark. Self-hosted runners work the same way.

The product mention is optional plumbing. Remove it and the allowlist-plus-overlap method is unchanged.

Limitations

Literal overlap is a heuristic. Shared status codes, HTTP ports, and protocol magic numbers create false positives; raise --max-overlap only with a written exception list, not by gut feel. The regex does not understand C++ user-defined literals, raw strings, or values built with constexpr math, so a determined model can still encode 1764 as 42 * 42 in production code. That encoding is why the hidden oracle exists.

Path allowlists miss tests registered from generated files, file(GLOB) in CMake, and CTest labels assembled at configure time. They also miss a model that weakens an assert without changing the path: assert(square_ll(42) == 1764) becoming assert(true). Catching that requires a hash of the test tree, which is a separate pin, not this script.

The workflow does not prove functional correctness, exception safety, or absence of undefined behavior. It does not replace -fsanitize=address,undefined, property-based tests, or a human review of the diff. It only fails candidates that treated the visible fixture as an answer key.

Who should not use this approach

Skip the leak grader when the agent’s job is to write tests as well as production code. TDD-style agents must edit tests/; an allowlist on src/ alone will fail every honest run. Skip it for safety-certified codebases that already require independent verification evidence this script cannot produce. Skip it when the “literals” in tests are the public API (error code tables, wire constants). In that case overlap is the point, not a cheat.

Teams that only ship one golden input per function should also pause. The fixture in this article is the bug. Add property tests or a hidden oracle first. Then turn the overlap threshold down.

The original dashboard pass was not a compiler success. It was an evaluation design error: the scorer and the answer key lived in the same files the model could read and patch. Freeze the tests. Hash the visible literals. Keep a second oracle off the prompt. Score those checks before celebrating a green ctest.

Operators who already pin prompts and goldens can keep this script next to those pins. A free-server job is one place to run it. A local cmake tree is enough to try the dishonest square_ll example the same afternoon.

When Snapshots Leak: Metamorphic Grading for AI C++ Patches

Finley Li — Sun, 20 Sep 2026 11:09:19 +0000

A C++ maintainer shipped an AI-authored merge_intervals patch on a Friday. Every golden file matched, and the CI badge stayed green. On Monday a caller passed the same ranges in a different order and the function returned overlapping pairs. The snapshots had all been pre-sorted. The model had learned the files, not the contract.

Static goldens still matter. They pin historical bugs. They do not survive a suite that leaked into prompts, gists, or fine-tunes, and they do not survive a generator that always emits the same shape of input. A relation grader treats those snapshots as a floor, then checks properties that stay true when the concrete lists change.

This article builds a small C++ eval harness around that split. It does not publish a leaderboard. It shows one way to catch silent failures that snapshot diffs miss.

What a metamorphic oracle actually checks

A snapshot says: for this exact input, emit this exact output. A metamorphic relation says: if the input is transformed in a known way, the outputs must stand in a known relationship. The second check does not need a stored answer key. It needs a generator, a transformation, and a comparison.

For interval merge, four relations are enough to start:

Permutation invariance: shuffling the input list must not change the merged set.
Idempotence: merging an already merged list must be a no-op.
Measure preservation: the total covered length after merge equals the size of the union of the originals.
Covering: every integer covered by an input interval is covered by some output interval, and the reverse.

A patch can fake (1) by sorting internally and still fail (3) if it drops a range. It can satisfy (3) with a sloppy union and still fail (4) if it emits a gap. The relations overlap on purpose.

Layout of the harness

Keep the candidate implementation separate from the oracle. The model may edit one translation unit. The driver, the generator, and the snapshot files stay outside the writable tree.

eval/
  include/merge_intervals.hpp
  candidate/merge_intervals.cpp
  oracle/metamorphic_driver.cpp
  goldens/case_001.in
  goldens/case_001.out
  tools/grade.sh
  seeds/current.seed

The header is the contract. It stays short on purpose.

#pragma once
#include <utility>
#include <vector>

// Merges overlapping and touching [lo, hi) intervals.
// Empty input yields an empty vector. Output is sorted by lo and pairwise disjoint.
std::vector<std::pair<int, int>> merge_intervals(
    std::vector<std::pair<int, int>> intervals);

A snapshot-only candidate that never sorts, and only walks the list in given order, will pass goldens that happen to be sorted. The metamorphic driver will not.

Step 1 — Pin three historical goldens

Goldens remain. They document bugs that already escaped once. They are not the whole spec.

goldens/case_001.in:

goldens/case_001.out:

2
1 6
8 10

Two further cases cover empty input and a single interval. The grader compares canonicalized text. Canonicalization sorts output pairs and rejects overlapping output as a snapshot failure too. That check is cheap. It is also leakable. Do not stop here.

Step 2 — Build a seeded generator

The generator must be deterministic given a seed, and cheap to rotate. A 64-bit integer in seeds/current.seed is enough. Changing the seed changes the concrete lists without changing the relations.

// oracle/metamorphic_driver.cpp (excerpt)
#include "merge_intervals.hpp"

#include <algorithm>
#include <cstdint>
#include <iostream>
#include <random>
#include <set>
#include <string>
#include <vector>

using Iv = std::pair<int, int>;

static std::vector<Iv> gen_list(std::mt19937_64& rng, int n) {
    std::uniform_int_distribution<int> pos(-50, 50);
    std::vector<Iv> out;
    out.reserve(static_cast<size_t>(n));
    for (int i = 0; i < n; ++i) {
        int a = pos(rng);
        int b = pos(rng);
        if (a > b) std::swap(a, b);
        if (a == b) ++b;  // keep half-open intervals non-empty
        out.push_back({a, b});
    }
    return out;
}

The distribution is narrow so union measure stays inside a small integer set. Production suites should widen the domain and accumulate coverage in 64-bit arithmetic. This file is a teaching artifact, not a fuzzer budget.

Step 3 — Encode the four relations

Each relation returns a short reason on failure and an empty string on success. The driver fails closed. Any non-empty reason is a grade of zero.

static std::vector<Iv> normalize(std::vector<Iv> v) {
    std::sort(v.begin(), v.end());
    return v;
}

static bool same_set(std::vector<Iv> a, std::vector<Iv> b) {
    return normalize(std::move(a)) == normalize(std::move(b));
}

static std::set<int> cover_points(const std::vector<Iv>& v) {
    std::set<int> pts;
    for (auto [lo, hi] : v) {
        for (int x = lo; x < hi; ++x) pts.insert(x);
    }
    return pts;
}

static std::string check_one(std::vector<Iv> in, std::mt19937_64& rng) {
    auto once = merge_intervals(in);
    auto twice = merge_intervals(once);
    if (!same_set(once, twice))
        return "idempotence";

    auto shuffled = in;
    std::shuffle(shuffled.begin(), shuffled.end(), rng);
    if (!same_set(merge_intervals(shuffled), once))
        return "permutation";

    if (static_cast<long long>(cover_points(once).size()) !=
        static_cast<long long>(cover_points(in).size()))
        return "measure";

    if (cover_points(once) != cover_points(in))
        return "cover";

    for (size_t i = 1; i < once.size(); ++i) {
        if (once[i].first < once[i - 1].second) return "overlap_in_output";
        if (once[i].first < once[i - 1].first) return "unsorted_output";
    }
    return {};
}

cover_points enumerates integers in a tiny domain. That is intentional. It makes the oracle obviously correct. It does not scale to 10^9 coordinates. A sweep-line oracle belongs in a later gate, not in this first relation file.

A complete main keeps golden mode and relation mode in one binary so the compiler sees the same candidate object file for both stages.

int main(int argc, char** argv) {
    std::uint64_t seed = 1;
    int trials = 200;
    std::string golden;
    for (int i = 1; i < argc; ++i) {
        std::string a = argv[i];
        if (a == "--seed" && i + 1 < argc) seed = std::stoull(argv[++i]);
        else if (a == "--trials" && i + 1 < argc) trials = std::stoi(argv[++i]);
        else if (a == "--golden" && i + 1 < argc) golden = argv[++i];
    }

    if (!golden.empty()) {
        int n = 0;
        if (!(std::cin >> n) && n != 0) return 2;
        std::vector<Iv> in(static_cast<size_t>(n));
        for (int i = 0; i < n; ++i) std::cin >> in[i].first >> in[i].second;
        auto out = merge_intervals(in);
        std::cout << out.size() << "\n";
        for (auto [lo, hi] : out) std::cout << lo << " " << hi << "\n";
        return 0;
    }

    std::mt19937_64 rng(seed);
    for (int t = 0; t < trials; ++t) {
        int n = static_cast<int>(rng() % 8);
        auto in = gen_list(rng, n);
        auto reason = check_one(in, rng);
        if (!reason.empty()) {
            std::cerr << "FAIL relation=" << reason << " trial=" << t << "\n";
            for (auto [lo, hi] : in) std::cerr << lo << " " << hi << "\n";
            return 1;
        }
    }
    return 0;
}

Silent assert macros disappear under some build flags. A printed relation name is the grade the eval operator actually reads.

Step 4 — Grade snapshots, then relations

The shell grader compiles the candidate, runs goldens, then runs the driver with the current seed. A green snapshot stage never skips the second stage.

#!/usr/bin/env bash
# tools/grade.sh
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
CXX="${CXX:-c++}"
CXXFLAGS="-std=c++17 -O1 -Wall -Wextra -Werror"
SEED="$(cat "$ROOT/seeds/current.seed")"
mkdir -p "$ROOT/build"

$CXX $CXXFLAGS -I "$ROOT/include" \
  "$ROOT/candidate/merge_intervals.cpp" \
  "$ROOT/oracle/metamorphic_driver.cpp" \
  -o "$ROOT/build/driver"

for case in "$ROOT/goldens"/*.in; do
  base="${case%.in}"
  "$ROOT/build/driver" --golden "$case" < "$case" > "$ROOT/build/got.out"
  diff -u "${base}.out" "$ROOT/build/got.out"
done

"$ROOT/build/driver" --seed "$SEED" --trials 200
echo "PASS seed=$SEED"

Store the seed next to the grade, not only in CI logs. A failing run that cannot be replayed is not an eval. It is a rumor.

Step 5 — Rotate the seed without rewriting the spec

Eval leakage is an engineering problem, not a slogan. Once a prompt, a golden file, or a public gist exists, a later model can echo the answers. Rotating the seed keeps the relations stable and the concrete cases fresh. A one-line write is the whole policy.

python3 - <<'PY'
import secrets
open("eval/seeds/current.seed", "w").write(str(secrets.randbits(64)))
PY

Replay uses the failed seed, not a new one. Rotation belongs between published eval batches, not between a crash and the debug rerun.

Snapshot vs relation vs sanitizer

The three gates catch different lies. Run them in that order so a compile error is not mistaken for a relation miss.

Gate	Catches	Misses
Historical snapshots	Exact bugs that already shipped	Unseen input shapes, leaked answer keys
Metamorphic relations	Order dependence, dropped coverage, non-idempotent merge	Values outside the generator domain
ASan/UBSan	Out-of-bounds, signed overflow, use-after-free	Wrong-but-defined output

No percentage in that table is a model score. It is a reminder of what each gate cannot see.

A patch that fools snapshots

The following candidate is the Friday patch. It assumes sorted input. It is short, and it is wrong.

#include "merge_intervals.hpp"

std::vector<std::pair<int, int>> merge_intervals(
    std::vector<std::pair<int, int>> intervals) {
    if (intervals.empty()) return {};
    std::vector<std::pair<int, int>> out{intervals.front()};
    for (size_t i = 1; i < intervals.size(); ++i) {
        if (intervals[i].first <= out.back().second)
            out.back().second = std::max(out.back().second, intervals[i].second);
        else
            out.push_back(intervals[i]);
    }
    return out;
}

Against case_001 this returns 1 6 and 8 10. The snapshot stage is green. Against a shuffled list the permutation relation fires. That is the entire point of stage B. No coverage percentage is required to see it.

A correct candidate sorts first, then sweeps. The relations do not encode that algorithm. They only reject results that violate the contract. Two different correct sweeps should both pass.

Where a hosted codegen path fits

A local compiler remains the grader. The model only proposes candidate/merge_intervals.cpp. When the eval operator wants that proposal loop without standing up a GPU box, MonkeyCode's free model access and free server option can host the generate-and-grade cycle. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The harness above does not depend on that host. It runs under any c++ that speaks C++17. Product quotas, model names, and hardware are out of scope here because they change, and because the grade is a compiler result, not a vendor score.

The useful split is simple. Generation may be remote. Judgment stays local: compile, snapshot diff, relation driver, recorded seed.

Limitations

The integer cover-set oracle is O(width × n). It is a teaching device. It will not grade 64-bit map ranges. Floating-point intervals need a different measure. Touching versus overlapping is a spec choice; this header treats touching as mergeable because hi is exclusive and a.hi == b.lo means adjacent coverage. A library that wants a gap there must change the relation, not the seed.

Metamorphic checks do not replace negative compilation tests, sanitizer matrices, or ABI dumps. They also do not prove partial correctness. A function that always returns the empty vector fails measure and cover, but a function that merges correctly on this generator's narrow distribution can still fail on INT_MIN. Widen the generator before treating a pass as evidence.

Seed rotation does not defeat a model that has learned the relations themselves. If the prompt pastes this article into context, the candidate may sort, then sweep, then pass. That is a real ceiling. Held-out relations, a second-language shadow oracle, and human review of failing lists are the next layers. This file does not pretend to include them.

Duplicate empty ranges can hide if an oracle canonicalizes too aggressively. The normalize helper above sorts pairs; it does not drop zeros, because empty [lo, lo) intervals are already forbidden by the generator. An API that allows empty ranges needs an extra relation that states that policy in one place.

Who should not use this approach

Do not use this harness as a production fuzzer for untrusted patches that must not run on the eval host. The driver executes candidate code. Sandbox that process separately. Do not use cover-point enumeration on large domains. Do not treat a PASS on 200 trials as a proof. Do not drop historical goldens; relations miss some exact-output regressions that snapshots catch, such as an extra diagnostic interval that still covers the same points after a sloppy canonicalize.

Teams that only need a yes/no compile gate should stay on the compiler. Teams that already leak every golden into public prompts should rotate more than the seed: they should rotate the function under test.

The relation grader is a small gate, and it is honest about that. Operators who already compile patches on a free-server workflow can drop tools/grade.sh beside the candidate file and read PASS seed=... as the only score that matters.

Asserts at -O0, UB at -O2: A Flag-Matrix Grader for AI C++ Patches

Finley Li — Sat, 19 Sep 2026 09:23:24 +0000

A maintainer once accepted an AI-written patch to a small C++ ring buffer. The eval job compiled the translation unit with the toolchain default, which in that repo meant -O0 -g. Three golden cases passed. The pull request looked boring in the best way.

The release build used -O2. Under wraparound the buffer dropped the newest element and left the oldest one intact. Nothing in the goldens had failed. The compiler had been allowed to assume that a signed index never overflowed, and the optimized code deleted the path the tests thought they were covering.

That pattern now shows up in AI C++ evals with uncomfortable regularity. Models play the harness they can see. A harness that only ever builds at -O0 grades a different program than the one that ships.

Why a single debug build is not a spec

Golden asserts answer one question: did this binary, built this way, produce these outputs. They do not answer whether the source still has a defined meaning under another flag set. Undefined behavior is the gap between those two measurements. At -O0 many UB bugs still appear to work. At -O2 the same bugs become deleted branches, hoisted loads, or loops that never terminate.

Sanitizers close part of that gap. They are not a second test suite. They are a different observer on the same goldens. An eval that records only the exit status of a debug binary is grading the observer, not the patch.

The rest of this article proposes a local flag-matrix grader. It rebuilds each candidate patch under several compiler envelopes, reruns the same goldens, and fails the patch if any envelope disagrees. The method is small enough to run on a laptop. It does not replace code review.

What the matrix actually measures

The grader treats a patch as a tuple of outcomes, not a boolean. Each cell is one compiler envelope:

-O0 -g — the historical eval default.
-O2 -DNDEBUG — a typical release envelope.
-O0 -fsanitize=address,undefined — ASan plus UBSan on the goldens.
-O2 -fsanitize=undefined — UBSan after the optimizer has rewritten the IR.

A pass means every golden still exits 0, stdout matches the pinned fixture, and the sanitizer log is empty. A fail in any cell fails the patch. Disagreement across cells is itself a finding. A patch that is green at -O0 and red at -O2 is not flaky. It is under-specified.

This is adjacent to coverage gates and resource envelopes, but it is not the same measurement. Coverage asks whether the hunk ran. rusage asks how fat the process got. The flag matrix asks whether the program still has a defined meaning when the compiler is allowed to believe the language.

A subject small enough to grade

The example below is a proposed local harness, not a measured benchmark. It encodes a realistic AI mistake: mixing a signed counter with a size computation that can overflow.

// subject/ring.hpp
#pragma once
#include <cstddef>
#include <cstdint>
#include <vector>

struct Ring {
  std::vector<std::uint32_t> buf;
  int head = 0;   // signed on purpose
  int count = 0;

  explicit Ring(int cap) : buf(static_cast<std::size_t>(cap), 0) {}

  void push(std::uint32_t v) {
    int cap = static_cast<int>(buf.size());
    int idx = head + count;
    if (idx >= cap) idx -= cap;   // breaks if head+count overflows
    buf[static_cast<std::size_t>(idx)] = v;
    if (count < cap) ++count;
    else head = (head + 1) % cap;
  }

  std::uint32_t front() const {
    return buf[static_cast<std::size_t>(head)];
  }
};

A model asked to make push faster will sometimes replace the wrap with a bitmask, or widen only half the arithmetic. Goldens that stay near small cap values never tickle signed overflow. The -O2 cell and the UBSan cell do.

// eval/golden_ring.cpp
#include "ring.hpp"
#include <cstdio>

int main() {
  Ring r(8);
  for (int i = 0; i < 20; ++i) r.push(static_cast<std::uint32_t>(i));
  if (r.front() != 12u) {
    std::fprintf(stderr, "front=%u\n", r.front());
    return 1;
  }
  return 0;
}

That golden is honest and still weak. Weak goldens are the point. The matrix exists to make their weakness visible instead of hiding it behind one debug binary.

Pin the envelopes in a manifest

Compiler flags drift the same way prompts drift. A one-line CXXFLAGS in a README is not a pin. Put the matrix in a file the grader hashes and commit that file next to the goldens.

{
  "cxx": "c++",
  "std": "c++17",
  "goldens": ["eval/golden_ring.cpp"],
  "include": ["subject"],
  "envelopes": [
    {"name": "dbg",    "cxxflags": ["-O0", "-g", "-Wall", "-Wextra", "-Werror"]},
    {"name": "rel",    "cxxflags": ["-O2", "-DNDEBUG", "-Wall", "-Wextra", "-Werror"]},
    {"name": "asan",   "cxxflags": ["-O0", "-g", "-fsanitize=address,undefined", "-fno-omit-frame-pointer"]},
    {"name": "ubsan2", "cxxflags": ["-O2", "-fsanitize=undefined", "-fno-sanitize-recover=undefined"]}
  ]
}

-Werror belongs in the debug and release cells. AI patches often introduce unused helpers and silent fallthrough that debug builds swallow if warnings are only advisory. The sanitizer cells omit -Werror only when a given toolchain warns about sanitizer-incompatible flags. Record that exception in the manifest. Do not hide it in a wrapper script.

Grader: rebuild, rerun, compare

The script below is a proposed driver. It compiles each golden under each envelope, runs the binary, and writes a JSONL row per cell. It does not scrape vendor dashboards. The grade happens on the machine that produced the binary.

#!/usr/bin/env python3
"""Proposed flag-matrix grader for AI C++ patches. Unexecuted example."""
from __future__ import annotations

import hashlib, json, os, subprocess, sys, tempfile
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
MANIFEST = json.loads((ROOT / "eval" / "matrix.json").read_text())

def run(cmd, extra_env=None, timeout=20):
    env = os.environ.copy()
    if extra_env:
        env.update(extra_env)
    return subprocess.run(
        cmd, capture_output=True, text=True, timeout=timeout, env=env
    )

def compile_one(src: Path, out: Path, flags: list[str]) -> subprocess.CompletedProcess:
    cmd = [
        MANIFEST["cxx"], f"-std={MANIFEST['std']}",
        "-I", str(ROOT / MANIFEST["include"][0]),
        *flags, str(src), "-o", str(out),
    ]
    return run(cmd)

def main() -> int:
    patch_id = sys.argv[1] if len(sys.argv) > 1 else "local"
    rows = []
    fail = 0
    with tempfile.TemporaryDirectory() as td:
        td = Path(td)
        for golden in MANIFEST["goldens"]:
            src = ROOT / golden
            for envp in MANIFEST["envelopes"]:
                out = td / f"{Path(golden).stem}_{envp['name']}"
                c = compile_one(src, out, envp["cxxflags"])
                cell = {
                    "patch": patch_id,
                    "golden": golden,
                    "envelope": envp["name"],
                    "compile_rc": c.returncode,
                    "compile_err": c.stderr[-2000:],
                }
                if c.returncode != 0:
                    cell["status"] = "compile_fail"
                    fail += 1
                    rows.append(cell)
                    continue
                extra = {}
                joined = " ".join(envp["cxxflags"])
                if "sanitize" in joined:
                    extra["UBSAN_OPTIONS"] = "print_stacktrace=1:halt_on_error=1"
                    extra["ASAN_OPTIONS"] = "detect_leaks=1:halt_on_error=1"
                r = run([str(out)], extra_env=extra)
                cell["run_rc"] = r.returncode
                cell["stderr"] = r.stderr[-2000:]
                cell["stdout_sha"] = hashlib.sha256(r.stdout.encode()).hexdigest()[:16]
                cell["status"] = "pass" if r.returncode == 0 and not r.stderr else "run_fail"
                if cell["status"] != "pass":
                    fail += 1
                rows.append(cell)
    outp = ROOT / "eval" / "last_matrix.jsonl"
    outp.write_text("".join(json.dumps(x) + "\n" for x in rows))
    print(f"wrote {outp} fail_cells={fail}")
    return 0 if fail == 0 else 2

if __name__ == "__main__":
    raise SystemExit(main())

Wire it with a short Make target so a human can reproduce one cell without Python.

CXX ?= c++
STD ?= c++17
GOLDEN ?= eval/golden_ring.cpp

dbg:
    $(CXX) -std=$(STD) -O0 -g -Wall -Wextra -Werror -I subject $(GOLDEN) -o /tmp/ring_dbg
    /tmp/ring_dbg

rel:
    $(CXX) -std=$(STD) -O2 -DNDEBUG -Wall -Wextra -Werror -I subject $(GOLDEN) -o /tmp/ring_rel
    /tmp/ring_rel

asan:
    $(CXX) -std=$(STD) -O0 -g -fsanitize=address,undefined -I subject $(GOLDEN) -o /tmp/ring_asan
    UBSAN_OPTIONS=halt_on_error=1 ASAN_OPTIONS=halt_on_error=1 /tmp/ring_asan

Run make dbg rel asan on the unpatched tree first. That is the noise floor for this measurement. If release already disagrees with debug on main, the matrix is grading the subject, not the model. Fix the subject before scoring patches.

How silent regressions show up

Read the JSONL as a table, not as a score. Four patterns matter.

All four cells pass. The goldens still may be weak. The matrix did not prove correctness. It proved agreement under these observers.
Debug passes, release fails. Treat this as UB or as a NDEBUG-dependent assert until proven otherwise. Do not rerun until green. Capture the stderr and the flag set.
Debug and release pass, ASan or UBSan fail. The goldens executed a bug the optimizer did not yet exploit. Keep the patch rejected.
Compile fails only under -Werror. That is a spec violation if the manifest pinned warnings as errors. Models pad patches with unused helpers. The compiler is the grader.

A useful extra column is stdout_sha. AI patches sometimes "fix" a golden by printing a different but still plausible line. Exit code 0 is not an I/O contract. Hash the bytes.

Disagreement is the signal worth keeping. A dashboard that averages the four cells into one pass-rate will hide the only interesting row. Store the tuple. Fail closed if any required cell is missing, not only if a cell is red.

Where generation stops and grading starts

Candidate patches have to come from somewhere. A local eval loop can call an open-source coding assistant, save the diff, and then refuse to trust that diff until the matrix is green.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode fits only as a generation step. The project is open source, with operator-supplied free-tier model access and a free server option that can emit candidate patches. The grade still belongs on a machine that has a C++ compiler, libasan, and the golden sources. Convenience is how -O0 defaults sneak back into remote notebooks. Keep matrix.json in the repository that owns the C++.

The workflow does not depend on that product. Any model that can emit a unified diff can sit in front of the same grader. A free server that writes diffs is useful. A compiler envelope that can reject them is the actual eval.

Limitations

Sanitizers do not see every UB. Data races need TSan and a concurrent golden. Strict-aliasing bugs sometimes need -fstrict-aliasing plus a release cell, not UBSan alone. -O2 is not -O3, and it is not LTO. A matrix with four cells is a sample of the compiler, not a proof.

The example golden is tiny. Real suites need the same envelopes applied to every binary the patch can touch, including tests the model did not edit. Link-time mismatches (-fsanitize on one translation unit, not on another) produce false stories. The grader above compiles one file. A CMake project needs the envelopes as preset cache files, not as ad-hoc c++ lines.

Wall time grows linearly with cells. Four envelopes times thirty goldens is a different job than one debug binary. Cache object files by (patch_sha, envelope_name) or the loop will train people to skip it. Skipping it returns the harness to the original lie: debug green equals shipable C++.

Who should not use this

Teams without a local C++ toolchain should not pretend a web playground is a release envelope. Firmware trees that cannot link sanitizer runtimes should keep the -O0 / -O2 disagreement cells and drop ASan, rather than skip the matrix. Safety-certified codebases need their own qualified compilers; this grader is not a qualification artifact.

Maintainers hunting model leaderboard points will hate the extra red cells. That is the point. Do not use the matrix as a substitute for a negative-compilation spec, a coverage gate on the hunk, or a resource envelope. Those catch other lies. This one catches the lie that debug green equals defined C++.

A working order of operations

Record four envelopes in eval/matrix.json. Hash the file in the eval lock.
Run the matrix on main until every cell is quiet. That is the baseline, not a score.
Generate a candidate patch from whatever model is in use.
Apply the patch on a worktree. Refuse to merge if any cell changes from pass to fail.
If debug and release disagree, stop and read sanitizer traces before asking the model to try again.
Promote a new golden only when it fails at least one envelope on a known-bad patch and passes all envelopes on main.

The order is boring on purpose. Eval harnesses fail when the interesting part is the model and the compiler is a default. Green asserts at -O0 are a beginning. They are not a ship signal. The flag matrix is a cheap way to ask the compiler the question the goldens forgot.

MonkeyCode provides free models that can run this workflow.

Dropped Exports, Green Asserts: Grading AI C++ Patches With nm

Finley Li — Fri, 18 Sep 2026 07:23:33 +0000

A plugin vendor filed a crash report that was not a crash. Their process started, then dlopen failed with undefined symbol: _ZN4mesh6Loader9fromFileERKSs. The library under test had shipped a green CI run two days earlier. An AI-authored patch had “simplified” a loader by making fromFile an inline header helper and leaving a new fromPath overload for the unit tests to call.

Asserts passed. The .so no longer exported the symbol the plugin had linked against. Functional goldens do not describe linkage. Coverage on a changed hunk does not describe linkage either. The missing check was an export list.

What a symbol golden actually grades

A symbol golden is a sorted list of mangled names the shared object must still provide after the patch. The grader builds the library, dumps dynamic symbols, and diffs that set against the list. Extra exports can be warnings. Missing exports are failures.

Renames show up as one deletion and one addition. The report prints them as a pair so a reviewer does not treat a rename as two unrelated events. Mangled names are the source of truth. Demangled text is for humans only.

c++filt output changes with the filt implementation. The committed golden therefore stores nm names, not prose. This article treats the export set as part of the eval spec for AI C++ patches, next to compile and test. It does not replace libabigail. It catches a class of silent regressions that unit tests reward.

A compact fixture

The fixture is a two-file shared library plus one plugin-shaped client. The client is not run as a unit test. It exists to make the failure mode obvious when the export disappears.

include/mesh/loader.hpp

#pragma once
#include <string>
#include <vector>

namespace mesh {

#if defined(MESH_BUILDING)
#define MESH_API __attribute__((visibility("default")))
#else
#define MESH_API
#endif

class MESH_API Loader {
public:
  static std::vector<unsigned char> fromFile(const std::string& path);
  static std::vector<unsigned char> fromPath(const std::string& path);
};

} // namespace mesh

src/loader.cpp

#define MESH_BUILDING
#include "mesh/loader.hpp"
#include <fstream>
#include <iterator>
#include <stdexcept>

namespace mesh {

std::vector<unsigned char> Loader::fromFile(const std::string& path) {
  std::ifstream in(path, std::ios::binary);
  if (!in) throw std::runtime_error("open failed");
  return std::vector<unsigned char>(
      std::istreambuf_iterator<char>(in),
      std::istreambuf_iterator<char>());
}

std::vector<unsigned char> Loader::fromPath(const std::string& path) {
  return fromFile(path);
}

} // namespace mesh

test/test_loader.cpp — the naive eval that stays green after an ABI break

#include "mesh/loader.hpp"
#include <cassert>
#include <filesystem>
#include <fstream>

int main() {
  auto p = std::filesystem::temp_directory_path() / "mesh_eval.bin";
  {
    std::ofstream out(p, std::ios::binary);
    out.put('A');
  }
  auto bytes = mesh::Loader::fromPath(p.string());
  assert(bytes.size() == 1);
  assert(bytes[0] == 'A');
  return 0;
}

A model that deletes fromFile from the .cpp and marks it inline in the header still satisfies this main. Downstream dlopen does not. That split is the whole reason to grade the .so.

Numbered eval workflow

Build the unpatched tree as a shared object with hidden visibility by default and explicit default on the API.
Dump dynamic defined symbols with nm -D --defined-only --format=posix.
Keep only names that start with _Z, plus any documented C wrapper prefix.
Sort uniquely and store the list as goldens/libmesh.so.syms.
Apply the candidate patch, rebuild with the same flags, and dump again.
Fail the eval on any missing golden symbol. Record extras separately so a human can promote them.

Hidden visibility matters. Without -fvisibility=hidden, every template instantiation becomes noise. The golden then rots on the first STL bump, and the harness starts accusing every model of ABI breakage.

Build flags the grader must pin

The compile line is part of the spec. Changing -fvisibility or C++ ABI flags makes two symbol dumps incomparable. Pin the compiler, the language dialect, and the soname in the same manifest that pins the eval prompt.

CXX ?= g++
CXXFLAGS += -std=c++17 -O2 -fPIC -fvisibility=hidden -fvisibility-inlines-hidden
LDFLAGS_SO += -shared -Wl,-soname,libmesh.so.1

libmesh.so.1: src/loader.cpp include/mesh/loader.hpp
    $(CXX) $(CXXFLAGS) -I include -o $@ src/loader.cpp $(LDFLAGS_SO)

test_loader: test/test_loader.cpp libmesh.so.1
    $(CXX) $(CXXFLAGS) -I include -o $@ test/test_loader.cpp -L. -lmesh \
      -Wl,-rpath,'$$ORIGIN'

A later flag drift is a harness bug. It is not a model regression. Rebuild HEAD against the golden before scoring any patch.

Dump and grade

tools/dump_syms.sh

#!/usr/bin/env bash
set -euo pipefail
so="${1:?shared object}"
# posix format: name type value size
nm -D --defined-only --format=posix "$so" \
  | awk '$2 ~ /^[TDBW]$/ { print $1 }' \
  | grep -E '^_Z' \
  | sort -u

T, D, B, and W cover text, initialized data, BSS, and weak defaults that plugins actually bind to. Undefined symbols (U) stay out. Absolute debug noise stays out. The awk column test is the filter. Do not pipe nm through c++filt before the golden diff.

tools/grade_syms.py

#!/usr/bin/env python3
"""Compare dumped dynamic symbols against a committed golden list."""
from __future__ import annotations

import argparse
import pathlib
import sys


def load_list(path: pathlib.Path) -> list[str]:
    lines = [
        ln.strip()
        for ln in path.read_text(encoding="utf-8").splitlines()
        if ln.strip() and not ln.lstrip().startswith("#")
    ]
    return sorted(set(lines))


def main() -> int:
    p = argparse.ArgumentParser(description="Symbol-set grader for C++ patch evals")
    p.add_argument("--golden", type=pathlib.Path, required=True)
    p.add_argument("--actual", type=pathlib.Path, required=True)
    p.add_argument("--allow-extra", action="store_true")
    args = p.parse_args()

    golden = load_list(args.golden)
    actual = load_list(args.actual)
    missing = [s for s in golden if s not in actual]
    extra = [s for s in actual if s not in golden]

    if missing:
        print("FAIL missing exports:")
        for s in missing:
            print(f"  - {s}")
    if extra:
        label = "NOTE extra exports:" if args.allow_extra else "FAIL extra exports:"
        print(label)
        for s in extra:
            print(f"  + {s}")

    if missing:
        return 2
    if extra and not args.allow_extra:
        return 3
    print("PASS symbol golden")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Run order after a candidate patch:

chmod +x tools/dump_syms.sh tools/grade_syms.py
make clean && make libmesh.so.1
./tools/dump_syms.sh ./libmesh.so.1 > /tmp/actual.syms
python3 tools/grade_syms.py \
  --golden goldens/libmesh.so.syms \
  --actual /tmp/actual.syms

Exit 2 means the patch deleted or renamed a public entry. Exit 3 means the surface grew. Growth is not always wrong. It is a different decision from “tests passed.”

Score the triple, not a single boolean

Folding compile, tests, and symbols into one pass/fail bit hides the failure mode. AI patch evals need the mode. The table below is a decision matrix, not a measured leaderboard.

compile	tests	symbols	interpretation
fail	n/a	n/a	does not build; stop
pass	fail	pass	functional miss
pass	pass	fail	silent ABI break
pass	fail	fail	mixed miss; inspect both diffs
pass	pass	pass	candidate, not a ship decision

Labeled example, unexecuted on a public corpus: a patch that inlines fromFile and keeps fromPath for tests yields compile=pass, tests=pass, symbols=fail. That triple is the point of the harness.

Wiring the grader into an AI patch eval

The eval loop stays boring on purpose.

Check out the base commit and confirm the golden still matches HEAD. If it does not, the harness is broken, not the model.
Apply the model patch as a single diff. Reject patches that touch goldens/libmesh.so.syms unless the task prompt explicitly asks for an API change.
Rebuild. If the link of libmesh.so.1 fails, record a compile/link miss, not an ABI miss.
Run test_loader. A test fail is a functional miss.
Dump and grade symbols. A symbol miss is an ABI miss even when step 4 passed.
Store the triple (compile, tests, symbols) as the score. Do not OR the bits into one trophy.

Step 1 is the sanity check for this envelope. A golden that already mismatches HEAD will accuse every model of ABI breakage. Step 2 stops the model from “fixing” the eval by rewriting the spec. Both steps are cheaper than another round of prompt tweaking.

Projects that already freeze ABI with a linker version script can dump that file instead of raw nm output. The grader then diffs the global: names. The idea is the same. The noise floor is lower because internal instantiations never enter the list.

LIBMESH_1.0 {
  global:
    _ZN4mesh6Loader8fromFileERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE;
    _ZN4mesh6Loader8fromPathERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE;
  local:
    *;
};

Commit the map. Pass --version-script at link time. Teach dump_syms.sh to fail if a golden name is present in the map but absent from nm. Two sources of truth that disagree are a harness incident.

Where a free codegen host fits

Patch generation still needs a model endpoint. Local grading still needs a Linux box with g++ and nm.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode advertises free model access and a free server option. Those two claims are the only product facts used here. This workflow does not depend on a particular model name, quota, or GPU SKU. A free server is useful only when the same Makefile, the same nm flags, and the same golden file run there and on the maintainer laptop.

If the remote image ships a different libstdc++ or a different default visibility, the symbol set is a different spec. The eval then measures the image, not the patch. Teams that already self-host models can keep generation local and still use the grader unchanged. The artifact is the symbol list and the exit codes, not the vendor.

Limitations

The method assumes ELF shared objects and GNU or LLVM nm posix output. Static libraries, Apple Mach-O, and Windows PE need different dumpers. LTO and --gc-sections can drop a symbol that a non-LTO golden still lists. Inline namespaces, version scripts, and the std::string dual ABI (__cxx11) make goldens toolchain-specific.

Template-heavy headers export whatever got instantiated in the .so. That set is not a stable API. Projects in that shape should golden a version script instead of the raw dump. The grader in this article does not check sizeof, field layout, or exception specifications. abidiff covers that layer. This envelope is the cheap filter that unit tests skip.

Do not treat a passing symbol golden as proof that plugins will load. dlopen still cares about RPATH, soname, and dependent .so files. Those belong in a separate loader smoke test. Mixing them into grade_syms.py blurs the failure mode the table above exists to keep distinct.

Who should skip this

Header-only libraries have no .so to dump. Applications that never export a plugin ABI gain little. Eval suites that already run a full abidiff gate do not need a second, weaker list. Windows-only shops should not pretend nm -D is portable.

Do not use this as a reason to freeze every internal helper. Visibility defaults exist so the golden can stay small. If the dump contains thousands of std:: instantiations, fix the flags before blaming the model. A noisy golden is not stricter. It is less reproducible.

Closing

AI C++ evals that only score asserts will keep accepting patches that move the public surface. A sorted symbol file is a dull artifact. It is also a reproducible one: same compiler, same visibility, same nm format, same exit codes.

Keep the golden next to the tests. Refuse patches that edit it without an explicit API task. Promote extras only when a human intends a new export. Clone the scripts, break fromFile on purpose, and confirm exit 2 before scoring any model output. A hosted generator is optional, and only honest if it repeats that same dump.

Zero Hits on the Hunk: A Coverage Gate for AI C++ Patch Evals

Finley Li — Thu, 17 Sep 2026 05:31:01 +0000

A C++ service accepted an AI patch on a Friday afternoon. The golden suite stayed green. The reviewer saw a plausible rewrite of decode_frame() and merged.

Monday's on-call page was a truncated header from a partner. decode_frame() had never run under the goldens. Those cases only constructed payloads that skipped the new branch. The harness had measured a function the patch did not touch.

This failure mode is not a flaky test. It is a blind one. When models outgrow the cases used to score them, a green golden file can mean the suite never called the hunk.

Blind goldens are a different bug class

Output digests catch silent changes in printed bytes. Resource envelopes catch RSS spikes. Flag matrices catch -O2 surprises. None of those checks ask whether any golden executed the lines the model rewrote.

Coverage is a weak proxy for correctness. It is a strong proxy for contact. A hunk with zero hits was not evaluated. Treating that result as a pass is a category error.

The rest of this article is a small, reproducible envelope. It fails the eval when patched executable lines report zero hits under the existing goldens. The method is a harness design, not a published benchmark.

Decision table before any flags

The gate is a predicate on two facts the CI already has: the golden result, and hit counts on the diff.

Golden result	Scored hunk lines	Envelope
red	any	reject; the suite already failed
green	every listed line count > 0	coverage accept only
green	some listed line counts = 0	reject
green	no executable patched lines scored	reject; suite and patch did not overlap

A coverage accept is not a merge. Digest checks, sanitizers, and must-not-compile cases still apply. This table only fills the hole where the suite never touched the rewrite.

Artifact: a coverage envelope

The envelope has five steps. Each fragment can be copied into a CI job. Replace the sample paths with the repository under test.

1. Record the hunk, not the whole tree

Coverage of untouched files is noise. The gate should score the patch.

#!/usr/bin/env bash
# record_hunk.sh — example workflow, not a measured production run
set -euo pipefail
BASE="${1:-origin/main}"
git diff -U0 "$BASE"...HEAD -- '*.cpp' '*.cc' '*.cxx' '*.h' '*.hpp' \
  > /tmp/patch.diff

python3 - <<'PY'
import re, json
hunks = {}
path = None
for line in open("/tmp/patch.diff"):
    if line.startswith("+++ b/"):
        path = line[6:].strip()
        hunks.setdefault(path, [])
        continue
    m = re.match(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@", line)
    if m and path:
        start = int(m.group(1))
        length = int(m.group(2) or "1")
        hunks[path].extend(range(start, start + length))
json.dump(hunks, open("/tmp/hunk_lines.json", "w"), indent=2)
print("wrote /tmp/hunk_lines.json")
PY

The JSON map is the only input the grader trusts for what changed. Untracked generated files stay out of the map unless the operator adds them on purpose.

2. Rebuild with instrumentation

Compile the test binary the same way the goldens already compile it, plus coverage flags. Mixing -O2 and --coverage is allowed, but the flag set should stay pinned. A coverage build is a different translation. Do not compare its timings against a release binary.

cmake -S . -B build-cov \
  -DCMAKE_BUILD_TYPE=Debug \
  -DCMAKE_CXX_FLAGS="--coverage -fno-inline -fno-elide-constructors"
cmake --build build-cov -j"$(nproc)"

-fno-inline keeps hunk lines from vanishing into a caller. That flag is a measurement choice. It is not a claim that inlining is unsafe in production.

3. Run the existing goldens, nothing extra

Do not add tests in this step. The point is to measure the suite that already claims to accept the patch.

cd build-cov
find . -name '*.gcda' -delete
ctest --output-on-failure

If the goldens fail, the coverage gate does not run. A red suite is already a reject. The envelope only speaks when the goldens are green.

4. Map hits onto the patch

GCC can emit gcov JSON. Parse that, not the text reports. GCC 9 and later accept --json-format.

# from the build tree, after goldens
while IFS= read -r gcno; do
  gcov -b -c --json-format "$gcno" >/dev/null
done < <(find . -name '*.gcno')

A compact grader then joins hunk_lines.json with the gcov JSON.

# cov_gate.py — example grader, not an executed leaderboard
from __future__ import annotations

import json
import sys
from pathlib import Path

def load_gcov(dir: Path) -> dict[str, dict[int, int]]:
    hits: dict[str, dict[int, int]] = {}
    for p in dir.rglob("*.gcov.json"):
        data = json.loads(p.read_text())
        for f in data.get("files", []):
            src = f["file"]
            line_hits = hits.setdefault(src, {})
            for ln in f.get("lines", []):
                line_hits[int(ln["line_number"])] = int(ln.get("count", 0))
    return hits

def basename_match(hunk_path: str, gcov_path: str) -> bool:
    return Path(hunk_path).name == Path(gcov_path).name

def main() -> int:
    hunks = json.loads(Path("/tmp/hunk_lines.json").read_text())
    hits = load_gcov(Path("."))
    missed = []
    scored = 0
    for path, lines in hunks.items():
        file_hits = {}
        for gpath, lh in hits.items():
            if basename_match(path, gpath) or gpath.endswith(path):
                file_hits = lh
                break
        for n in lines:
            # skip lines gcov never lists (comments, braces-only)
            if n not in file_hits:
                continue
            scored += 1
            if file_hits[n] == 0:
                missed.append(f"{path}:{n}")
    report = {
        "scored_executable_lines": scored,
        "zero_hit_lines": missed[:50],
        "zero_hit_count": len(missed),
    }
    Path("/tmp/cov_gate.json").write_text(json.dumps(report, indent=2))
    if scored == 0:
        print("cov_gate: no executable patched lines were scored", file=sys.stderr)
        return 2
    if missed:
        print(f"cov_gate: {len(missed)} patched lines had zero hits", file=sys.stderr)
        return 1
    print("cov_gate: every scored patched line was hit")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Exit code 2 means the suite and the patch did not overlap at all. Exit code 1 means some patched executable lines stayed cold. Both are rejects under this envelope.

5. Fail closed, then keep the report

Store /tmp/cov_gate.json next to the golden digest. A later prompt pin or model swap can compare zero-hit counts the same way it compares hashes. The coverage gate is a predicate, not a leaderboard.

A toy program makes the trap obvious.

// src/codec.cpp
int decode_body(const char* p, int n) { return n > 0 ? p[0] : -1; }

int decode_frame(const char* p, int n) {
    if (n < 2) return -1;
    return (static_cast<unsigned char>(p[0]) << 8) |
           static_cast<unsigned char>(p[1]);
}

// tests/golden_codec.cpp
#include <cassert>
int decode_body(const char*, int);
int decode_frame(const char*, int);
int main() {
    const char body[] = "x";
    assert(decode_body(body, 1) == 'x');
    return 0;
}

An AI patch that "simplifies" decode_frame still leaves this golden green. The coverage gate reports zero hits on that function and rejects the eval. That is the entire point.

How this relates to models that outgrow their tests

Public debate in mid-September 2026 returned to a familiar claim: coding models already beat most developers, or that passing a test suite is the same as engineering. Those slogans are not a spec. A green golden file is evidence that the cases ran. It is not evidence that the cases touched the patch.

When a model learns the suite, it can keep outputs stable while moving behavior into untested functions. Coverage-on-the-hunk is a cheap tripwire for that pattern. It does not stop a model from writing a wrong line that a golden does execute. It stops the quieter failure: scoring a rewrite the suite never called.

Where a spare model host fits

Generating candidate patches and rebuilding with --coverage is CPU work. Local laptops throttle. A shared box that already compiles the goldens is enough.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. Those two facts are the only product claims used here. This article does not name models, quote a token quota, or report a latency number. The coverage grader above runs on any Linux host with GCC, CMake, and Python 3. If the eval already lives in that toolchain, the free server option is one place to park overnight coverage gates without changing the predicate.

What this envelope does not prove

Coverage contact is not functional correctness. A line can run and still be wrong.

Inlined templates, constexpr evaluation, and macros can make gcov line numbers disagree with the diff. Basename matching is a convenience and can collide. Header-only libraries need -fkeep-inline-functions or the hunk disappears.

Death tests and exec tests may not write .gcda files. Parallel ctest without GCOV_PREFIX can clobber counters. The grader must run after a single, serialized golden pass or use per-test prefixes.

A 100% hit rate on the hunk can still hide a removed check. Digest comparison and negative-compilation cases remain necessary. This gate only fills the never-called hole.

Who should skip this gate

Skip it when the patch is comments, CMake, or pure data files. gcov will not score those paths, and a zero scored_executable_lines result is expected.

Skip it when the team already runs mutation testing on the same goldens. Mutation is stronger and more expensive. The coverage gate is the cheaper filter.

Skip it when the binary cannot be rebuilt with --coverage, including some vendor toolchains and freestanding firmware without a host test double.

Do not use the zero-hit count as a model ranking. It is a boolean envelope on a single patch. Ranking models by coverage invites the model to insert dead code that a later pass covers with a trivial call.

A pinned command the CI can copy

#!/usr/bin/env bash
set -euo pipefail
./record_hunk.sh origin/main
cmake -S . -B build-cov -DCMAKE_BUILD_TYPE=Debug \
  -DCMAKE_CXX_FLAGS="--coverage -fno-inline"
cmake --build build-cov -j"$(nproc)"
( cd build-cov && find . -name '*.gcda' -delete && ctest --output-on-failure )
( cd build-cov && find . -name '*.gcno' -print0 | xargs -0 -n1 gcov -b --json-format >/dev/null )
python3 cov_gate.py

Pin the compiler version next to this script. Coverage IR changes across GCC releases. A gate that flips because the image moved from GCC 12 to GCC 13 is a harness bug, not a model regression.

The Friday merge in the opening would have failed step 5. The goldens would still have been green. The report would have listed src/codec.cpp lines inside decode_frame as zero-hit. That is a small, boring, and sufficient reason to reject the eval.

Same Goldens, New Question: A Prompt-Pin Manifest for AI C++ Evals

Finley Li — Wed, 16 Sep 2026 03:48:53 +0000

A C++ library team merged a “clearer” system prompt on a Thursday. The golden cases stayed green. The compiler stayed quiet. A reviewer opened the next model patch on Monday and found using namespace std; in a public header, dropped const on three accessors, and a helper that threw from a destructor path the tests never entered.

Nothing in the scorecard had moved. The tests still asked last month’s question. The prompt no longer did.

Golden cases measure the answer. They do not measure whether the question is still the same question. Prompt edits are dependency changes. An eval that does not pin the prompt will report stability while the instruction surface drifts.

This article describes a small eval identity for AI C++ patches. The harness refuses to score a candidate unless a manifest of hashes matches the files on disk. The method stays useful if every product name is stripped out. A later section notes where a free remote runner can sit in the loop.

What the scorecard usually forgets

Most AI C++ harnesses version three things: the source under test, the compiler flags, and the expected I/O. That is necessary. It is not sufficient.

A prompt rewrite can keep every assert passing while changing which defects the model is even asked to avoid. Style rules vanish. Exception-safety language is softened. The model is told to prefer brevity. The goldens still check that add(2, 2) returns 4.

Silent question drift looks like a quality win. It is a spec change without a bump. Leaderboard rows from two prompt texts are then treated as one experiment. They are not.

Eval identity, not a vibe

Treat the eval as a tuple. If any element changes, the run is a different experiment.

Prompt text, including the system message and the user template.
Golden cases: inputs, expected outputs, and skip flags.
Grader: compiler invocation, sanitizers, and assertion driver.
Toolchain pins: compiler version string, language standard, and flag vector.
Candidate patch: the model output under test.

Hash items 1–4 into a committed manifest. Compare those hashes before any score is written. Fail closed on mismatch. Record the candidate hash only after the identity check passes.

The rest of this article implements that check with ordinary files. No database. No dashboard. The scripts below are a proposed tree, not a claim that they have been run against a private corpus.

Artifact: `eval_manifest.json`

Store the manifest next to the prompt, not in a wiki. Empty hashes are placeholders. A freeze script fills them. CI rejects a run whose live hashes do not match the committed pins.

{
  "schema": "eval-identity/v1",
  "suite": "ai-cpp-patches",
  "pins": {
    "prompt_sha256": "",
    "goldens_sha256": "",
    "grader_sha256": "",
    "toolchain_sha256": ""
  },
  "toolchain": {
    "cxx": "g++",
    "cxx_std": "c++20",
    "cxxflags": ["-O2", "-Wall", "-Wextra", "-Werror", "-fsanitize=address,undefined"]
  },
  "policy": {
    "on_mismatch": "fail_closed",
    "allow_prompt_bump": false
  }
}

allow_prompt_bump stays false in CI. A human freeze is the only way the prompt pin moves. That pause is the whole gate.

Layout on disk

Keep the suite boring and grepable. One prompt file. One golden directory. One grader tree. One captured toolchain dump.

eval/
  prompt.txt
  goldens/
    001_add.json
    002_overflow.json
    003_header_hygiene.json
  grader/
    run.sh
    catch_main.cpp
  toolchain.txt
  eval_manifest.json
  freeze.py
  check_identity.py
  score.py

toolchain.txt is a captured g++ -v dump plus the exact flag vector. Do not reconstruct it from memory during a run. Two machines with “the same g++” still diverge on patch level, default search paths, and sanitizer runtimes.

Step 1 — Freeze the identity

freeze.py hashes canonical bytes. The script does not pretty-print JSON before hashing goldens. It hashes the files as committed, in sorted path order, so a renamed case still moves the tree digest.

#!/usr/bin/env python3
"""Freeze eval identity. Proposed helper; run it on a real tree before trusting pins."""
from __future__ import annotations

import hashlib
import json
from pathlib import Path

ROOT = Path(__file__).resolve().parent

def sha256_file(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()

def sha256_tree(dirpath: Path) -> str:
    h = hashlib.sha256()
    for p in sorted(dirpath.rglob("*")):
        if not p.is_file():
            continue
        rel = p.relative_to(dirpath).as_posix().encode()
        h.update(rel)
        h.update(b"\0")
        h.update(p.read_bytes())
        h.update(b"\0")
    return h.hexdigest()

def main() -> None:
    man_path = ROOT / "eval_manifest.json"
    man = json.loads(man_path.read_text())
    man["pins"]["prompt_sha256"] = sha256_file(ROOT / "prompt.txt")
    man["pins"]["goldens_sha256"] = sha256_tree(ROOT / "goldens")
    man["pins"]["grader_sha256"] = sha256_tree(ROOT / "grader")
    man["pins"]["toolchain_sha256"] = sha256_file(ROOT / "toolchain.txt")
    man_path.write_text(json.dumps(man, indent=2) + "\n")
    print("froze", man["pins"])

if __name__ == "__main__":
    main()

Run it only when a spec bump is intended.

python3 eval/freeze.py
git add eval/eval_manifest.json eval/prompt.txt eval/goldens eval/grader eval/toolchain.txt
git diff --cached eval/eval_manifest.json

The pin diff is the changelog. If the prompt hash moved and the goldens hash did not, the question changed and the tests did not. That pair is the reason the harness exists.

Step 2 — Fail closed before any model call

check_identity.py recomputes live hashes and compares. Scoring never starts on mismatch. Generation can wait. A wrong question is cheaper to catch before tokens are spent than after a green cell is published.

#!/usr/bin/env python3
from __future__ import annotations

import json
import sys
from pathlib import Path
from freeze import ROOT, sha256_file, sha256_tree

def live_pins() -> dict[str, str]:
    return {
        "prompt_sha256": sha256_file(ROOT / "prompt.txt"),
        "goldens_sha256": sha256_tree(ROOT / "goldens"),
        "grader_sha256": sha256_tree(ROOT / "grader"),
        "toolchain_sha256": sha256_file(ROOT / "toolchain.txt"),
    }

def main() -> int:
    man = json.loads((ROOT / "eval_manifest.json").read_text())
    expected = man["pins"]
    got = live_pins()
    bad = {
        k: {"expected": expected[k], "got": got[k]}
        for k in expected
        if expected[k] != got[k]
    }
    if bad:
        print("EVAL_IDENTITY_MISMATCH")
        print(json.dumps(bad, indent=2))
        return 2
    print("EVAL_IDENTITY_OK", json.dumps(got, sort_keys=True))
    return 0

if __name__ == "__main__":
    sys.exit(main())

Wire it as a gate, not as a log line.

python3 eval/check_identity.py && python3 eval/score.py --patch candidate.diff

A typical mismatch looks like this after someone “just clarified” prompt.txt:

EVAL_IDENTITY_MISMATCH
{
  "prompt_sha256": {
    "expected": "9c1e0c6c0f3a7b1d8a2e4f55c0b91aa0e3c6d2f1a7b84c0d9e1f2a3b4c5d6e7f",
    "got":      "0b77a1d4e2c98f06b3aa4419d5c0e18f27d6b4c1a9e03f55c8d2a176b0e4c9aa"
  }
}

Those hex strings are examples of shape, not measurements from a private run. The important part is the exit code. CI treats 2 as a broken spec, not as a model failure.

Step 3 — Keep goldens that the new question can still fail

Prompt pins without hostile goldens only detect file drift. They do not detect a weaker instruction. Add at least one golden that encodes a rule the prompt claims to enforce. Header hygiene is a reliable example because tests almost never look at it.

goldens/003_header_hygiene.json:

{
  "id": "003_header_hygiene",
  "source": "box.hpp",
  "must_compile": true,
  "compile_unit": "const Box b{1}; auto x = b.value(); (void)x;",
  "forbid_regex": [
    "using namespace std;",
    "using namespace std::",
    "catch\\s*\\(\\s*\\.\\.\\.\\s*\\)"
  ]
}

A grader compiles the unit and scans the patch text. The regex layer is crude. It still catches the Thursday rewrite that stopped mentioning public headers. Put the scan in score.py so the grader tree hash moves when the check changes.

grader/run.sh stays the only compiler entry. That keeps the tree hash meaningful.

#!/usr/bin/env bash
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
python3 "$ROOT/check_identity.py"
CXX="${CXX:-g++}"
src="$1"
"$CXX" -std=c++20 -O2 -Wall -Wextra -Werror -fsanitize=address,undefined \
  -c "$src" -o /tmp/candidate.o

Capture the live toolchain once per machine image.

{
  echo "# toolchain pin"
  g++ -v 2>&1
  echo "# flags"
  echo "-std=c++20 -O2 -Wall -Wextra -Werror -fsanitize=address,undefined"
} > eval/toolchain.txt

Label the snippets as a proposed envelope. They are not a published benchmark and they are not a claim about any model’s pass rate.

Step 4 — Record the candidate only after identity holds

score.py should refuse to write last_result.json unless check_identity returned zero. Then hash the patch itself. Later spreadsheet rows become comparable only when the pin block matches.

# Proposed fragment, not a full product.
import hashlib, json, subprocess, sys
from pathlib import Path

patch_path = Path(sys.argv[1])
patch = patch_path.read_bytes()
subprocess.check_call([sys.executable, "eval/check_identity.py"])
pins = json.loads(Path("eval/eval_manifest.json").read_text())["pins"]
result = {
    "eval_manifest": pins,
    "patch_sha256": hashlib.sha256(patch).hexdigest(),
    "verdict": "unscored",
}
Path("eval/last_result.json").write_text(json.dumps(result, indent=2) + "\n")

Mixing rows from two prompt hashes is a methodology error. The file makes that error visible. A green cell under pin set A is not a data point under pin set B.

Decision table: what to bump

Change	Prompt pin	Goldens pin	Grader pin	Toolchain pin	Action
Wording tweak in `prompt.txt`	yes	no	no	no	Freeze, re-run the full suite, do not compare scores to the previous pin set
New golden for header hygiene	no	yes	maybe	no	Freeze, treat as spec expansion
Switch `-O0` to `-O2`	no	no	no	yes	Freeze; scores are a different experiment
ASan added to `run.sh`	no	no	yes	maybe	Freeze; old “green” is not transferable
Model sampled again, same files	no	no	no	no	Score and compare
Prompt and goldens both edited	yes	yes	no	no	Freeze once, write a note that two spec axes moved together

The dangerous row is the first one. Teams skip it because the tests still pass. The manifest exists to make that skip loud. Two axes moving in one commit is legal. It is also easy to misread later, so the note belongs in the same commit as the freeze.

Where a free remote runner fits

Local identity checks are cheap. Generating candidate patches is not. Operators who already iterate prompts against a C++ suite sometimes need a place to sample models without standing up a GPU box for every wording tweak.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project that, as supplied by the operator of this account, offers free model access and a free server option. Those two availability claims are the only product facts used here. This article does not assert model names, token quotas, hardware, uptime, or permanence. Dashboard numbers go stale. Re-read the project page before planning capacity.

In this workflow the remote runner is optional. The identity files stay in git. A generation job may run off-laptop. The score step still happens only after check_identity.py passes on the same commit. If the free server is used to produce candidate.diff, copy that file back and hash it locally. Do not let a remote workspace silently edit prompt.txt.

Readers who want that runner can look up the public repository when they need it. The harness above does not depend on it.

Limitations

The manifest does not understand semantics. Two prompts can hash differently and mean the same thing. Two prompts can hash identically after a well-placed Unicode lookalike. Hashing is identity, not equivalence. Operators who need semantic diff still have to read the prompt.

Regex forbid-lists are bypassable. A model can split using namespace std; across a line continuation or hide a throw in a macro. A single compile unit does not prove ABI stability, thread safety, or that RSS stayed flat. Those are other envelopes. They do not replace a prompt pin, and a prompt pin does not replace them.

This approach is a poor fit for throwaway playground prompts, for suites without committed goldens, and for teams that want a single scalar “model quality” number across prompt versions. It is the wrong tool if the compiler cannot be pinned. Unpinned toolchains make the toolchain hash theater.

Do not use the freeze script as a ritual that rubber-stamps a weaker prompt. If the prompt hash moved, re-read the goldens. Add a case that would have failed under the old instruction. Then freeze.

What “green” is allowed to mean

After this harness is in place, a green cell means four files on disk still match a committed tuple, and the candidate survived that tuple’s grader. It does not mean the prompt is better. It does not mean last week’s leaderboard row is comparable.

Prompt edits become visible spec bumps. Silent question drift has to walk through freeze.py. That is a small gate. For C++ codegen evals, it is usually the gate that was missing.

The Placebo Patch: Measure Your C++ Eval Harness's Noise Floor Before You Trust a Regression

Finley Li — Mon, 14 Sep 2026 21:34:38 +0000

A four-person team running an AI patch loop on a C++ service added a merge gate last quarter. One rule: if any golden case changes verdict, the candidate patch is rejected. On a Tuesday morning the gate rejected a patch that renamed buf_len to buffer_length in a single function.

Nothing changed at runtime. The rename moved one assertion message, one stack trace line, and the interleaving of two log lines. The harness reported three regressions, and all three were artifacts.

The gate was measuring itself. What it lacked was a control arm, and a control arm is cheap to build.

The number nobody records

Every eval harness has a noise floor: the rate at which verdicts change when nothing semantic changes. That number is a property of a triple — harness, toolchain, machine — and it moves whenever any leg moves.

Teams usually discover it the expensive way. A regression is reported, a developer spends a day bisecting, and the conclusion is "flaky case, rerun it." Then the gate loses its authority, and real regressions get rerun to death alongside the imaginary ones.

A placebo corpus fixes this in about eighty lines of glue.

Step 1 — Build a placebo corpus

A placebo patch is a semantically neutral edit whose only job is to expose harness brittleness. Five classes cover most of the ground.

ID	Placebo patch	What it perturbs
P0	null patch (identical tree)	pure run-to-run variance
P1	comment-only insertion	nothing intentional
P2	local variable rename in one TU	debug info, assertion messages, symbol layout
P3	reorder `#include` lines in one file	declaration order, token positions
P4	`clang-format` applied to one function	line and column numbers
P5	add an unused `static` helper	symbol table, link order

P0 is the honest baseline. P1 through P5 should be invisible; when they are not, the case is reacting to shape rather than behavior.

Store them as unified diffs under placebo/patches/, named so they sort predictably.

placebo/
  patches/
    00-null.patch        # empty diff, applied as a no-op
    01-comment.patch
    02-rename.patch
    03-include-order.patch
    04-format.patch
    05-dead-static.patch
  out/                   # one JSON per (patch, run)

Writing P2 and P4 by hand across a 2,000-line translation unit is dull work with a sharp oracle attached: if a supposedly neutral edit accidentally changes behavior, the first matrix run exposes it, not a reviewer. That self-verifying property is what makes the variant generation a reasonable job for free model access.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The harness in this setup generates its neutral variants through MonkeyCode's free model access. Quota, throughput, and availability terms belong to the project's own documentation and should be read there rather than from any number quoted in an article.

Step 2 — Run the matrix

The harness needs one property: it must emit machine-readable verdicts, one entry per case. A --json flag on stdout is enough.

#!/usr/bin/env bash
# placebo/run_matrix.sh
set -euo pipefail

K="${K:-5}"                       # runs per patch
OUT=placebo/out
mkdir -p "$OUT"

for patch in placebo/patches/*.patch; do
  name=$(basename "$patch" .patch)
  for run in $(seq 1 "$K"); do
    if [ "$name" != "00-null" ]; then
      git apply "$patch"
    fi
    cmake --build build -j"$(nproc)" >/dev/null
    ./build/harness --json > "$OUT/$name.$run.json"
    git checkout -- .             # drop the placebo patch
  done
done

Two details decide whether the data means anything. Build with the same compiler, flags, and machine for the whole matrix, because a mixed matrix measures the toolchain. Keep K odd if the report is going to talk about a modal verdict.

Step 3 — Turn verdicts into a per-case admission report

The matrix is only raw material. The useful output is one row per golden case with two booleans: did it disagree with itself across runs, and did it flip under a neutral patch?

#!/usr/bin/env python3
"""noise_report.py - admission verdict per golden case."""
import collections, glob, json, pathlib

REF_PATCH = "00-null"

by_patch = collections.defaultdict(list)
for path in sorted(glob.glob("placebo/out/*.json")):
    record = json.loads(pathlib.Path(path).read_text())
    by_patch[record["patch"]].append(record)

cases = sorted(next(iter(by_patch.values()))[0]["cases"])
report = {}

for case in cases:
    reference = by_patch[REF_PATCH][0]["cases"][case]["status"]

    flake = any(
        len({r["cases"][case]["status"] for r in runs}) > 1
        for runs in by_patch.values()
    )
    placebo_sensitive = any(
        reference not in {r["cases"][case]["status"] for r in runs}
        for patch, runs in by_patch.items()
        if patch != REF_PATCH
    )

    report[case] = {
        "flake": flake,
        "placebo_sensitive": placebo_sensitive,
        "admission": ("gate" if not flake and not placebo_sensitive
                      else "advisory" if not flake
                      else "quarantine"),
    }

pathlib.Path("placebo/noise_report.json").write_text(
    json.dumps(report, indent=2, sort_keys=True))
excluded = [c for c, r in report.items() if r["admission"] != "gate"]
print(f"{len(excluded)}/{len(cases)} cases excluded from the merge gate")

Three outcomes fall out of the two booleans, and each one has a different owner.

run-to-run flake	placebo sensitive	admission	action
no	no	`gate`	eligible to block merges
no	yes	`advisory`	normalize the observable, or stop blocking on it
yes	either	`quarantine`	fix or delete on a deadline, never gate

The middle row is what pays for the exercise. A case that never flakes but flips when a function is reformatted is not measuring behavior; it is measuring line numbers.

Step 4 — Gate on the promotable set only

The gate script stays deliberately small. It reports a regression only when the case is in the gate set, and it prints everything it suppressed.

#!/usr/bin/env python3
"""gate.py CANDIDATE.json REFERENCE.json"""
import json, sys

noise = json.load(open("placebo/noise_report.json"))
candidate = json.load(open(sys.argv[1]))
reference = json.load(open(sys.argv[2]))

gateable = {c for c, r in noise.items() if r["admission"] == "gate"}
changed = {
    c for c in candidate["cases"]
    if candidate["cases"][c]["status"] != reference["cases"][c]["status"]
}

print(json.dumps({
    "regressions": sorted(changed & gateable),
    "suppressed_noisy": sorted(changed - gateable),
}, indent=2))

sys.exit(1 if changed & gateable else 0)

suppressed_noisy still needs to be visible. A case that changed verdict but is not gateable is exactly the information a human needs to decide whether to repair the case instead of the patch.

Step 5 — Re-measure instead of inheriting

The noise floor is not a constant, so the report has an expiry date. Re-run the matrix when the compiler version, the harness, or the CI machine class changes.

A five-patch by five-run matrix multiplies suite time by twenty-five. That makes it a nightly job rather than a per-pull-request job. The operator runs it on MonkeyCode's free server option, which is a defensible use of idle cycles: the control arm is slow, deterministic, and produces nothing user-visible.

If the suite is too slow even for that, cut K to three and say so in the report. A wide interval that is labeled honestly beats a narrow one that is invented.

Limitations

The control arm measures variance, not correctness. If every golden case encodes the same wrong expectation, all placebo runs agree with each other, and the gate stays confidently wrong.

Networked or timing-dependent cases belong outside the placebo arm. Five runs over a live HTTP dependency measure the network's mood, not the harness's stability. Keep the corpus on deterministic unit-level cases.

The gateable set can shrink. Track its size over time. A rough rule of thumb: if the promotable set falls below about sixty percent of the corpus, the corpus needs rewriting rather than more quarantine.

Who should skip this

A solo project with one fast, deterministic suite and no automated patch loop gains little here, because there is no gate to defend.

Anyone without a reproducible reference build should fix that first. The reference run is the baseline that every comparison in the report is drawn against.

Teams whose individual cases take minutes cannot run P x K matrices at all. They should start with P0 alone and one run per patch, just to find out whether the floor is measurable on their hardware.

Closing

A gate that fires on a variable rename costs more trust than it buys. Measuring the floor takes one script, one nightly job, and a decision table, and it converts "flaky case, rerun it" into a number somebody can act on. Nothing in this workflow requires a paid runner — the whole experiment above was run on MonkeyCode's free tier, which is also a low-stakes way to find out whether the control arm earns a permanent slot in CI.

Must-Not-Compile Is a Spec: A Negative-Compilation Envelope for AI C++ Patches

Finley Li — Sun, 13 Sep 2026 19:52:52 +0000

A payments adapter review landed on a Tuesday afternoon. The model had produced a three-file diff. The unit suite returned green in twelve seconds, and the reviewer merged it.

A week later a plugin author constructed TokenHandle from a raw integer. The original header had blocked that path with an explicit private constructor and a deleted converting constructor. The patch had moved the constructor to public so a test the model added in the same diff would compile.

Runtime assertions never saw the leak. The type had become a wider type while every EXPECT_* still passed. Positive tests reward compilation. They do not record the compilation that must keep failing.

The hole in treating build success as progress

Most AI C++ patch evals stop at three bits. The compiler exit code is zero. The test binary exits zero. Sometimes stdout is hashed. That triad is a spec for “does something run.” It is not a spec for “the public surface did not grow.”

Language-level contracts live in the failures. Deleted special members, explicit constructors, private factories, requires clauses, and static_assert checks exist so that certain translation units never become programs. An edit that turns those failures into successes is a behavior change. Graders that treat every successful compile as progress will score that edit as a win.

The envelope below treats selected compile failures as first-class oracles. It is a proposed, reproducible harness. It is not a published leaderboard and it does not report model scores.

What the envelope records

A negative-compilation envelope is a directory of tiny translation units plus a grader. Each unit includes the public header under test and then performs one operation the API is supposed to reject.

The grader keeps three buckets:

Must-build units. These compile, link, and run. They are the ordinary happy path.
Must-fail units. The compiler must return a non-zero status. A diagnostic substring may be pinned.
Must-still-fail after the patch. If a must-fail unit starts compiling, the patch is a surface regression even when runtime tests stay green.

The third bucket is the point of the harness. AI patches frequently unstick a test by weakening a type.

Verdict table

must-build after patch	must-fail after patch	Verdict
compiles and tests pass	still fails to compile	accept
compiles and tests pass	now compiles	surface regression
no longer compiles	still fails	patch broke the build
runtime assertion fails	still fails	functional regression
compiles and tests pass	fails, but diagnostic text drifted	toolchain mismatch, not a model win

A minimal library under test

The example is a move-only handle. Copy is deleted. Integer construction is not part of the public surface. The snippets are labeled examples, not extracted production code.

// include/token_handle.hpp
#pragma once
#include <cstdint>
#include <utility>

class TokenHandle {
public:
  static TokenHandle from_trusted(std::uint64_t id) noexcept {
    return TokenHandle(id);
  }

  TokenHandle(const TokenHandle&) = delete;
  TokenHandle& operator=(const TokenHandle&) = delete;

  TokenHandle(TokenHandle&& other) noexcept : id_(other.id_) {
    other.id_ = 0;
  }

  TokenHandle& operator=(TokenHandle&& other) noexcept {
    if (this != &other) {
      id_ = other.id_;
      other.id_ = 0;
    }
    return *this;
  }

  std::uint64_t id() const noexcept { return id_; }

private:
  explicit TokenHandle(std::uint64_t id) noexcept : id_(id) {}
  std::uint64_t id_;
};

A positive unit that must keep compiling and running:

// cases/pos_move_ok.cpp
#include "token_handle.hpp"
#include <cassert>

int main() {
  auto a = TokenHandle::from_trusted(42);
  auto b = std::move(a);
  assert(b.id() == 42);
  return 0;
}

Two negative units that must keep failing. These files are not broken tests. They are the contract.

// cases/neg_copy.cpp
#include "token_handle.hpp"

int main() {
  auto a = TokenHandle::from_trusted(1);
  TokenHandle b = a; // copy must remain deleted
  (void)b;
}

// cases/neg_from_int.cpp
#include "token_handle.hpp"

int main() {
  TokenHandle h = 7; // converting construction must remain impossible
  (void)h;
}

Manifest and grader

Pin the compiler, the language mode, and the include path. A JSON manifest keeps the envelope reviewable without extra YAML dependencies.

{
  "compiler": ["g++", "-std=c++20", "-Wall", "-Wextra", "-Werror"],
  "include": "include",
  "must_build": ["cases/pos_move_ok.cpp"],
  "must_fail": [
    {"path": "cases/neg_copy.cpp", "diagnostic": "deleted"},
    {"path": "cases/neg_from_int.cpp", "diagnostic": ""}
  ]
}

Proposed grader in Python 3. It shells out to the real compiler. It does not parse C++.

#!/usr/bin/env python3
"""Negative-compilation envelope. Proposed example, not a published score."""
from __future__ import annotations

import json
import subprocess
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parent
BIN = Path("/tmp/negcomp_a.out")


def compile_unit(compiler, include, src):
    cmd = compiler + [f"-I{include}", str(src), "-o", str(BIN)]
    return subprocess.run(cmd, capture_output=True, text=True)


def main():
    spec = json.loads((ROOT / "envelope.json").read_text())
    compiler = spec["compiler"]
    include = str(ROOT / spec["include"])
    failures = []

    for rel in spec["must_build"]:
        proc = compile_unit(compiler, include, ROOT / rel)
        if proc.returncode != 0:
            failures.append("must_build failed: " + rel + "\n" + proc.stderr)
            continue
        run = subprocess.run([str(BIN)], capture_output=True, text=True)
        if run.returncode != 0:
            failures.append("runtime failed: " + rel)

    for item in spec["must_fail"]:
        proc = compile_unit(compiler, include, ROOT / item["path"])
        if proc.returncode == 0:
            failures.append("SURFACE REGRESSION: " + item["path"] + " now compiles")
            continue
        needle = item.get("diagnostic") or ""
        blob = (proc.stderr or "") + (proc.stdout or "")
        if needle and needle not in blob:
            failures.append(
                "must_fail, diagnostic missed " + repr(needle) + ": " + item["path"]
            )

    if failures:
        print("\n".join(failures))
        return 1
    print("envelope: must-build green, must-fail still failing")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Run the grader before and after a candidate patch:

python3 grade_envelope.py
git apply /tmp/model.patch
python3 grade_envelope.py; echo "grade_exit=$?"
git checkout -- include/token_handle.hpp

A model that publishes the constructor to satisfy neg_from_int.cpp will flip that unit from fail to success. The second run prints SURFACE REGRESSION. Exit code zero on the unit tests is then irrelevant.

Numbered workflow for an AI C++ patch eval

Freeze envelope.json in the same commit as the headers it describes. Do not let the model edit the must-fail list in the same turn as the production sources.
Generate the patch in an isolated worktree. Keep envelope files read-only to the agent, or reject diffs that touch cases/neg_*.cpp.
Apply the patch. Rebuild must-build units. Run them.
Recompile every must-fail unit. Require a non-zero compiler status. If a diagnostic substring is pinned, require it.
Accept the patch only when all three hold: must-build still builds, tests still pass, must-fail still fails.
Record g++ --version or clang++ --version next to the envelope. Diagnostic text is not portable across major versions.

Step 2 is the lock. Oracle suites fail when the model rewrites the oracle. Negative units fail the same way if they are writable.

Where generation and grading split

Generating the candidate patch and compiling the envelope are separate jobs. The first is a language-model call. The second is a compiler farm.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free-tier model access can sit on the generate side of that split, and its free server option can sit on the grade side so the must-fail matrix does not occupy a laptop that is already compiling the product. Those are availability options, not quotas, model names, or latency claims. Teams that already pin a CI image should keep grading inside that image. The envelope does not depend on a particular vendor.

Maintainers who already score AI C++ patches can drop envelope.json onto an idle runner and compare pre-patch and post-patch compiler statuses. That is the only product-shaped suggestion in this write-up.

Compiler notes that break naive graders

GCC and Clang do not emit the same words for a deleted constructor. Pinning diagnostic to "deleted" can fail on a compiler that says use of deleted function versus one that says attempting to reference a deleted function. Status-only checks are more portable and weaker.

CMake try_compile is an alternative to the Python loop. The inversion is easy to misread in review.

try_compile(NEG_COPY_BUILT
  ${CMAKE_BINARY_DIR}/neg_copy
  SOURCES ${CMAKE_SOURCE_DIR}/cases/neg_copy.cpp
  CMAKE_FLAGS "-DINCLUDE_DIRECTORIES=${CMAKE_SOURCE_DIR}/include"
)
if(NEG_COPY_BUILT)
  message(FATAL_ERROR "neg_copy compiled; public surface widened")
endif()

try_compile returns true on success. The envelope treats true as failure. The Python grader makes that inversion obvious in one if proc.returncode == 0.

Concepts and SFINAE change the picture again. A constraint failure may be a substitution failure inside a template, not a hard error at top-level main. Negative units should call the constrained function in a context that forces a hard error. Otherwise the unit compiles and the envelope lies.

Limitations

The envelope does not prove API stability. It proves that a finite list of snippets still fail. A model can widen the surface along an unlisted axis: a new converting operator, a friend declaration, or a defaulted copy after a refactor into a trivially copyable layout.

Diagnostic substring matching is brittle across compiler upgrades. Status-only matching is coarse. Both need a pinned toolchain.

Compile-fail tests do not catch runtime undefined behavior that still compiles. They do not replace sanitizers, ABI dumps, or human review.

Header changes that only affect inlining can still be correct while flipping a must-fail unit if the unit was over-specified. Treat false positives as envelope bugs until a human reads the diagnostic.

Who should not use this

Do not adopt the envelope for patches that never touch headers or type definitions. A leaf .cpp bugfix has no compile-fail contract to protect.

Do not treat it as a security control. A private constructor is not a sandbox.

Do not run a single-compiler manifest as the only gate on a codebase that ships three toolchains. Split envelopes per compiler, or check status only.

Skip it when the team cannot keep cases/neg_*.cpp out of the model’s writable set. A model that deletes the negative units will score perfectly.

Closing

Runtime green is a weak statement about C++. The language spends a large fraction of its design budget on programs that must not exist. An eval that never asks the compiler to refuse those programs will systematically prefer models that widen types until the tests compile.

Keep the must-fail list small, versioned, and unwritable to the agent. Grade the patch against that list with the same compiler the project ships. The merge that only looks at ctest will keep promoting public constructors that were never in the spec.

MonkeyCode provides free models that can run this workflow.

Asserts Passed, RSS Doubled: An rusage Envelope for AI C++ Patches

Finley Li — Sat, 12 Sep 2026 17:51:49 +0000

The following incident is a composite of a common failure mode, not a claim about a named production system. A daemon team merged a one-line cache suggested by a coding model. Every unit test stayed green. The next on-call graph told a different story: resident set size climbed through the night, and a previously quiet process started swapping.

The patch replaced a linear scan with an unbounded unordered_map. Correctness tests used fixtures of a few dozen keys. Nothing in the pipeline asked how much memory or CPU a correct answer was allowed to spend.

Functional goldens remain necessary. They are not a resource spec.

Cost is part of the contract

AI C++ patches fail in noisy ways and in quiet ways. Missing headers and invented helpers break the build. Resource regressions usually do not. The binary still links. Asserts still pass. The process simply gets heavier on every call.

An eval harness that only checks exit codes will score the heavy patch as a win. A second artifact has to sit beside the tests: a committed resource envelope. The envelope is JSON next to the fixtures. It records ceilings, not averages. If a candidate exceeds a ceiling, the patch fails even when every assert passes.

This article is a worked example. It is not production telemetry. Numbers in the sample envelope are illustrative ceilings for a tiny driver. They are not measurements from a deployed service.

What the envelope records

The file stays small on purpose. Three fields catch a surprising number of helpful-looking model edits on Linux.

max_rss_kb — high-water resident set from child rusage (ru_maxrss, kilobytes on Linux).
max_user_us — user CPU time in microseconds.
timeout_s — wall-clock cap; a hang becomes a reject before RSS is even read.

Optional later fields include max_minflt and max_inblock. A field is worth adding only after a real incident names it. The envelope is a tripwire. Tripwires are allowed to be coarse.

Ceilings have to be recorded on the same class of host that will grade candidates. Copying a laptop envelope onto a shared eval box is a factory for false failures. Host class is part of the spec, the same way -O2 is part of the spec.

Worked example: the cache that never evicted

The sample library looks up scores by id. The baseline scans a small vector. Tests pass. A coding model then fixes the scan by filling a process-lifetime unordered_map with no eviction. Lookups still return the right integer. RSS does not stay still.

Header under test

#pragma once
#include <cstdint>
#include <unordered_map>
#include <vector>

struct ScoreRow {
  std::uint32_t id;
  int score;
};

inline int lookup_scan(const std::vector<ScoreRow>& rows, std::uint32_t id) {
  for (const auto& row : rows) {
    if (row.id == id) return row.score;
  }
  return -1;
}

// Anti-pattern used to demonstrate the envelope.
// Models emit this when a prompt says "make it faster" and never says "bound the table."
inline int lookup_unbounded_cache(const std::vector<ScoreRow>& rows,
                                  std::uint32_t id) {
  static std::unordered_map<std::uint32_t, int> cache;
  auto it = cache.find(id);
  if (it != cache.end()) return it->second;
  int score = lookup_scan(rows, id);
  cache.emplace(id, score);
  return score;
}

The unbounded cache is deliberate. It is the silent case: right answers, growing weight.

Driver the harness actually times

#include "score_lookup.hpp"
#include <cstdint>
#include <iostream>
#include <string>
#include <vector>

int main(int argc, char** argv) {
  const bool use_cache = (argc > 1 && std::string(argv[1]) == "cache");
  std::vector<ScoreRow> rows;
  rows.reserve(256);
  for (std::uint32_t i = 0; i < 256; ++i) {
    rows.push_back({i, static_cast<int>(i % 17)});
  }
  long long acc = 0;
  for (int round = 0; round < 40000; ++round) {
    std::uint32_t id = static_cast<std::uint32_t>((round * 17u) % 100000u);
    acc += use_cache ? lookup_unbounded_cache(rows, id)
                     : lookup_scan(rows, id);
  }
  std::cout << acc << "\n";
  return 0;
}

Both modes print one integer and exit 0. Functional equality is the trap. The envelope is the distinction.

Committed ceilings

{
  "cmd": ["./driver"],
  "timeout_s": 5,
  "max_rss_kb": 8192,
  "max_user_us": 800000,
  "notes": "Illustrative Linux ceilings for this driver. Re-record on the eval host."
}

Rebuild the envelope when the host, the compiler, or the fixture size changes. Treat that rebuild as a spec change. The grader should not raise ceilings on its own.

The rusage grader

The grader lives outside the candidate on purpose. A patch that prints RSS: ok is not evidence. Python 3 plus the resource module is enough on Linux. The parent process must not wait on other children before the timed run, or RUSAGE_CHILDREN will mix histories.

Label the script as a template. It is not certified on every distro. GNU /usr/bin/time -f '%x %U %M' is a valid alternative when Python is unwelcome. The committed artifact should still be envelope.json, not a screenshot of a terminal.

#!/usr/bin/env python3
"""Linux rusage envelope grader for C++ patch candidates. Worked example only."""
from __future__ import annotations

import argparse
import json
import resource
import subprocess
import sys
from pathlib import Path


def measure(cmd: list[str], timeout_s: float) -> dict:
    try:
        completed = subprocess.run(
            cmd,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            timeout=timeout_s,
            check=False,
        )
        timed_out = False
        returncode = int(completed.returncode)
        stdout = completed.stdout
        stderr = completed.stderr
    except subprocess.TimeoutExpired as exc:
        timed_out = True
        returncode = 124
        stdout = exc.stdout or b""
        stderr = exc.stderr or b""
    ru = resource.getrusage(resource.RUSAGE_CHILDREN)
    return {
        "timed_out": timed_out,
        "returncode": returncode,
        "stdout": stdout.decode("utf-8", errors="replace"),
        "stderr": stderr.decode("utf-8", errors="replace"),
        "max_rss_kb": int(ru.ru_maxrss),
        "user_us": int(ru.ru_utime * 1_000_000),
    }


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--cmd", nargs="+", required=True)
    parser.add_argument("--envelope", type=Path, required=True)
    parser.add_argument("--record", action="store_true")
    parser.add_argument("--force", action="store_true")
    parser.add_argument("--grade", action="store_true")
    parser.add_argument("--expect-stdout", type=Path)
    args = parser.parse_args()
    if args.record == args.grade:
        print("exactly one of --record or --grade is required", file=sys.stderr)
        return 2

    timeout_s = 5.0
    if args.grade:
        env = json.loads(args.envelope.read_text())
        timeout_s = float(env.get("timeout_s", 5))

    result = measure(args.cmd, timeout_s)

    if args.record:
        if args.envelope.exists() and not args.force:
            print("refusing to overwrite envelope without --force", file=sys.stderr)
            return 2
        # Slack is explicit in the file. Hidden slack inside the grader is a bug.
        payload = {
            "cmd": args.cmd,
            "timeout_s": timeout_s,
            "max_rss_kb": int(result["max_rss_kb"] * 1.3) + 1024,
            "max_user_us": int(result["user_us"] * 1.5) + 50_000,
            "notes": "ceilings = one baseline run plus committed slack; re-record on host change",
        }
        args.envelope.write_text(json.dumps(payload, indent=2) + "\n")
        print(json.dumps({"mode": "record", "measured": {
            "max_rss_kb": result["max_rss_kb"],
            "user_us": result["user_us"],
            "returncode": result["returncode"],
        }, "wrote": str(args.envelope)}))
        return 0

    env = json.loads(args.envelope.read_text())
    reasons = []
    if result["timed_out"]:
        reasons.append("timeout")
    if result["returncode"] != 0:
        reasons.append(f"exit_{result['returncode']}")
    if result["max_rss_kb"] > int(env["max_rss_kb"]):
        reasons.append("rss")
    if result["user_us"] > int(env["max_user_us"]):
        reasons.append("user_cpu")
    if args.expect_stdout and result["stdout"] != args.expect_stdout.read_text():
        reasons.append("stdout")
    verdict = "reject" if reasons else "accept"
    print(json.dumps({
        "mode": "grade",
        "verdict": verdict,
        "reasons": reasons,
        "measured": {
            "returncode": result["returncode"],
            "max_rss_kb": result["max_rss_kb"],
            "user_us": result["user_us"],
            "timed_out": result["timed_out"],
        },
    }, indent=2))
    return 1 if reasons else 0


if __name__ == "__main__":
    raise SystemExit(main())

Linux reports ru_maxrss in kilobytes. macOS reports bytes. This template is Linux-shaped on purpose. Mixing units without a platform tag will either reject every patch or accept every leak.

Numbered workflow

Run the loop the same way on every candidate. Drift in the loop is how silent cost bugs get a passing grade.

Pin the toolchain. Record g++ --version next to the envelope. A different libstdc++ can move RSS without any source change.
Build the known-good binary at -O2. Optimization level is part of the spec. Debug heaps lie about both RSS and user time.
Capture a baseline with --record. The record path should refuse to overwrite unless --force is passed.
Apply the candidate patch in a worktree. Rebuild with the same flags. Sanitizers stay out of this particular loop; they change RSS on purpose.
Grade the candidate with --grade and --expect-stdout. Read the JSON verdict line, not the driver's stdout, for the cost decision.
Reject when functional output mismatches or when any ceiling breaks. Accept only when both oracles agree.
If the cost increase is the actual design, raise envelope.json in the same commit as the patch. Split commits hide the trade.

A one-line compile plus the two harness modes looks like this:

set -euo pipefail
g++ --version
g++ -std=c++17 -O2 -Wall -Wextra -o driver driver.cpp
./driver > golden_stdout.txt
python3 rusage_harness.py --record --force --cmd ./driver --envelope envelope.json
python3 rusage_harness.py --grade --cmd ./driver cache \
  --envelope envelope.json --expect-stdout golden_stdout.txt

The interesting outcome is exit status 1 with "reasons": ["rss"] while stdout still matches. That is the patch that looks like competence in a unit-test UI.

Where generation and grading can share a host

Local laptops add noise. Browser tabs move max RSS. Thermal throttling moves user CPU. An eval that is supposed to catch silent weight needs a boring machine.

Teams that already generate C++ patches from models can run this envelope next to the compiler. MonkeyCode is an open-source project with free model access and a free server option, which is enough to keep both generation and the rusage loop off a developer workstation.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The envelope still has to live in the repository. Free inference does not define max_rss_kb. The server is a stable place to run the grader, not a substitute for committed ceilings. Model names, token quotas, and hardware details are out of scope here because they change and they are not what the envelope measures.

How the silent case shows up

The scan driver and the cache driver can share stdout. The harness then prints two different reasons to fail.

Observation	Functional oracle	Envelope oracle	Action
Same stdout, RSS and CPU under ceilings	pass	pass	accept
Different stdout, any cost	fail	ignored	reject
Same stdout, RSS over ceiling	pass	fail	reject as cost regression
Timeout	fail	fail	reject as hang
Compile error	fail	n/a	reject

The third row is the point of the extra artifact. Models also emit optimizations that raise CPU while lowering RSS, or the reverse. A single-field envelope will miss half of those. RSS and user CPU start together. They should not be averaged into one score. Averages hide which contract broke.

Limitations

ru_maxrss is a high-water mark, not a leak proof. A patch can allocate, free, and still pass if the peak stays under the ceiling. The envelope will not replace AddressSanitizer or a heap profiler.

Shared eval hosts are noisy. A noisy neighbor can raise minor faults and sometimes RSS. Pinning a cgroup, or committing a small slack percentage inside envelope.json, is honest. Hidden slack inside the Python script is a grader bug.

The driver above uses a static map, which is easy to see in review. Real regressions hide in extra copies of std::string, in std::regex constructed per call, in logging that formats on a hot path. The envelope does not name the guilty line. It only refuses to ship the binary.

Timeouts are wall-clock. A loaded host turns a legal patch into a hang. timeout_s needs the same honesty as RSS, and the machine should stay otherwise idle during grade. After TimeoutExpired, child rusage may be incomplete. Treat timeout as a hard reject rather than a partial RSS reading.

This loop does not prove asymptotic complexity. Forty thousand iterations can still miss an O(n²) path that production will hit at millions. Pair the envelope with at least one fixture that resembles production cardinality. Tiny goldens created the original incident.

Who should skip this approach

The technique does not replace perf or microbenchmark libraries when the question is a 2% inner-loop change. The tripwire is too coarse for that job.

It does not transfer to Windows without a different oracle. The script above is not a portable abstraction.

It does not belong on patches that are explicitly space-time tradeoffs unless the envelope update rides along in the same review. A grader that always loses to an intentional cache is theatre.

It should not run under extra sanitizers and then compare against a non-sanitized envelope. That comparison is meaningless.

Skip the technique when the binary's memory is dominated by a huge mmap the patch cannot affect. The ceiling will never move. The signal is zero.

Closing

Unit tests answer whether the answer was right. They do not answer whether the process was allowed to become larger while getting that answer. Commit both oracles. Grade both on every model patch.

The sample files above are small enough to drop beside an existing C++ eval. A stable host matters more than a clever dashboard. If patch generation already has spare capacity, envelope.json is the cheaper artifact to try on that same host first.