An editorial still life illustrating how superstitious prompt patches conceal critical structural failures beneath superficial semantic tape.

Picture this: It is 2:00 AM, and your flagship enterprise AI agent has just hallucinated a radical corporate discount policy for a tier-one client, triggering a massive, automated financial ledger discrepancy. Your lead engineer scrambles to the rescue, not by refactoring the codebase or analyzing state vectors, but by opening a chat window and appending a frantic, superstitious patch: “Crucial: Never, under any circumstances, invent discounts. Be very careful.” The deployment goes live. The bleeding stops momentarily. Everyone breathes a sigh of relief over lukewarm masala tea. But deep down, you know the truth. You haven’t fixed anything; you’ve just buried another toxic snippet of technical debt under a mountain of semantic quick-fixes.

📊 Executive Summary: Modern generative AI integration significantly spikes software delivery throughput while simultaneously acting as an institutional amplifier for underlying architectural instability (testmuai.com, 2024). Traditional software guarantees mathematically deterministic outputs from identical inputs (vincirufus.com, 2024), whereas stochastic AI agents rely on probabilistic reasoning (nintex.com, 2024). Managing this transition requires Enterprise AI Reliability Engineering (AIRe), utilizing decoupled control planes (elementum.ai, 2024), dual-indicator SLIs (testmuai.com, 2024; medium.com, 2024), uncertainty quantification (mit.edu, 2024), and quality-aware circuit breakers (bigyan.dev, 2024) to curb silent system degradation (nintex.com, 2024; ve3.global, 2024).

I. The Hook & The Big Promise

Since the dawn of assembly language, enterprise software engineering has operated on a comforting, mathematically absolute premise: a specific input reliably produces an identical, deterministic output (vincirufus.com, 2024). From financial core ledgers to regulatory compliance engines, our digital infrastructure was built on the rock-solid foundation of predictability (vincirufus.com, 2024; nintex.com, 2024). Enter Generative AI. We are now stitching probabilistic reasoning engines — systems that evaluate semantic likelihoods, weigh fluid context, and spit out stochastic variance — directly into the heart of mission-critical pipelines (nintex.com, 2024). While this integration provides an undeniable surge in delivery velocity, it also serves as a potent amplifier for deep architectural instability (testmuai.com, 2024).

“Superstitious prompt patches conceal architectural rot beneath layers of semantic illusion.” — Mohit Sewak

The industry’s default coping mechanism has been the “prompt hack” — treating system prompts and few-shot examples as magic strings used to patch symptoms on the fly. Engineering teams stitch together ad-hoc string formatting tricks while ignoring the silent structural degradation, unmanaged API rate limits, and catastrophic failure cascades brewing just beneath the application tier (ve3.global, 2024; testmuai.com, 2024). Treating a distributed, probabilistic state vector as a simple string variable is the architectural equivalent of holding up a crumbling skyscraper wall with masking tape.

A physical study contrasting rigid deterministic clockwork machinery with turbulent stochastic fluid dynamics, illustrating the vulnerability of enterprise pipelines.

This masterclass outlines Enterprise AI Reliability Engineering (AIRe) — a rigorous framework that adapts classical Site Reliability Engineering (SRE) principles to stochastic, non-deterministic environments (testmuai.com, 2024; medium.com, 2024). By the end of this blueprint, you will possess the operational models, mathematical uncertainty quantification frameworks, and code-level fault-tolerance patterns required to build production-grade, self-healing hybrid systems that survive contact with reality (testmuai.com, 2024; medium.com, 2024).

II. The Architectural Convergence: Marrying Deterministic Control Planes with Bounded Probabilistic Agents

The modern enterprise cannot choose between deterministic stability and probabilistic intelligence; it requires a disciplined hybrid approach (acceldata.io, 2024). Industry data indicates that only a small fraction of organizations achieve enterprise-wide LLM production, primarily because governance bottlenecks choke unconstrained deployments (stonebranch.com, 2024). To solve this, enterprise architecture is coalescing around the “symphony orchestra” model, which strictly segregates deterministic control from probabilistic execution (elementum.ai, 2024).

A physical architectural model illustrating the segregation of deterministic control planes from isolated probabilistic execution edge agents.

🔍 Fact Check: Only a small fraction of organizations achieve enterprise-wide LLM production due to unmanaged governance bottlenecks and architectural opacity.

In this hybrid topology, deterministic orchestration engines act as the strict conductor, managing distributed state, enforcing execution sequences, and maintaining immutable audit trails (elementum.ai, 2024; stonebranch.com, 2024). Probabilistic agents are securely relegated to the edges of the application to handle unstructured tasks like sentiment extraction, document parsing, and semantic synthesis (nintex.com, 2024; elementum.ai, 2024; kestra.io, 2024). Financial ledgers, compliance reporting, and loan origination pipelines demand absolute rule consistency, systematically excluding AI reasoning from final rule-execution steps (nintex.com, 2024; elementum.ai, 2024; kestra.io, 2024).

Data flows through strict interception checkpoints: the deterministic layer dictates the required JSON schema, and if a model’s output fails validation or dips below a confidence threshold, the control plane intercepts it before downstream propagation (elementum.ai, 2024; stonebranch.com, 2024; bigyan.dev, 2024). Never allow an unconstrained agent to orchestrate core business logic. If an identical input must equal an identical output for regulatory audits, bypass the probabilistic layer entirely (elementum.ai, 2024).

A laboratory balance and dual-column fluid manometer isolating infrastructure uptime from behavioral semantic fidelity and refusal ceilings.

III. Beyond “Three Nines”: Formulating Probabilistic SLAs, Refusal Ceilings, and Burn-Rate Freezes

Traditional SRE relies on binary availability metrics and uptime percentages like “three nines” (testmuai.com, 2024; medium.com, 2024). But an AI model can maintain 100% infrastructure uptime while completely failing semantically through hallucinations or toxic outputs (medium.com, 2024). AIRe resolves this by establishing a two-indicator system that isolates Infrastructure SLIs (computed deterministically at the gateway: TTFT, TPOT, HTTP error codes) from Behavioral SLIs (evaluated via programmatic validators and LLM-as-a-judge frameworks tracking semantic quality and prompt adherence) (testmuai.com, 2024; medium.com, 2024).

💡 ProTip: Isolate infrastructure uptime metrics from behavioral semantic validation using dual SLI indicators to prevent infrastructure health from masking silent model hallucinations.

To maintain granular visibility without over-provisioning expensive GPU clusters, state fields are tracked explicitly across categories like answered_primary, answered_fallback, answered_ungrounded, refused, and answered_after_retry (testmuai.com, 2024; researchgate.net, 2023, 2024). Crucially, a distinct “refusal ceiling” ensures that necessary safety guardrails rejecting malicious prompts do not unjustly drain the primary composite error budget (testmuai.com, 2024).

An optical test bench demonstrating how uncertainty quantification separates reducible epistemic variance from irreducible aleatoric ambiguity.

Alerts are mapped not to single-incident failures — which cause crippling alert fatigue in stochastic environments — but to error budget consumption velocity or “burn rates” (testmuai.com, 2024). When burn rates spike past acceptable thresholds, automated freeze policies immediately halt prompt edits, model upgrades, and tool integrations while permitting rollbacks and deterministic fallback routing (testmuai.com, 2024). Treat upstream vendor model deprecations or silent quality drops as direct hits to your internal error budget (testmuai.com, 2024).

IV. Runtime Self-Awareness: Operationalizing Uncertainty Quantification (UQ) and Decision Theory

To survive in production, an AI system must possess runtime self-awareness. Uncertainty Quantification (UQ) provides this by appending a numerical confidence score or probability distribution to model predictions (mit.edu, 2024; arxiv.org, 2021). UQ mathematically separates Epistemic Uncertainty (Model Uncertainty) — a lack of training data that is reducible via targeted fine-tuning or RAG — from Aleatoric Uncertainty (Data Uncertainty) — inherent input ambiguity that is completely irreducible (ibm.com, 2023; medium.com, 2023).

Industrial electrical switchgear and heavy steel bulkheads illustrating semantic circuit breakers and idempotent tooling.

Translating these variance metrics into business actions requires decision theory mapping using expected loss functions (Expected_Loss = ∫ Loss(y, a) P(y|x) dy) (medium.com, 2023). Rather than undertaking the massive computational expense of retraining primary foundation models, enterprise architectures deploy lightweight, low-latency companion evaluators that inspect outputs in real-time, compute precise uncertainty bounds, and trigger automated routing paths (mit.edu, 2024). When epistemic uncertainty is low but aleatoric uncertainty spikes due to a messy user prompt, bypass expensive generation loops and immediately trigger human-in-the-loop routing or deterministic clarification workflows (mit.edu, 2024; ibm.com, 2023).

V. Defensive Infrastructure: Quality-Aware Circuit Breakers, Model Bulkheads, and Idempotent Tooling

Distributed systems fail when external dependencies experience latency spikes or thread exhaustion (ve3.global, 2024; getmaxim.ai, 2024; microsoft.com, 2020). Intelligent retry logic must map explicitly to HTTP status codes, distinguishing transient 429 rate limits and 5xx server errors from persistent 400 malformed requests (getmaxim.ai, 2024; udemy.com, 2022). However, network-level circuit breakers are blind to semantic failure. An HTTP 200 OK carrying a hallucinated payload must trip a breaker just as quickly as a network timeout (bigyan.dev, 2024).

A terraced architectural staircase and decoupled overhead mechanical gantry modeling operational degradation ladders and task-decoupled planning.

Extended “Quality-Aware Circuit Breakers” integrate sliding quality windows, schema compliance validators, and recovery probes executing known-good inputs during the “Half-Open” state (bigyan.dev, 2024; microsoft.com, 2020). Simultaneously, Model Bulkheads enforce physical thread partitioning and asynchronous non-blocking IO wrappers, preventing slow-processing LLM endpoints from holding calling service thread pools hostage (researchgate.net, 2023; microsoft.com, 2020; studylib.net, 2014).

VI. Bounded Autonomy: Designing Degradation Ladders and Task-Decoupled Planning

When systems degrade, they must do so gracefully. Multi-agent workflows risk cascading failures where an error at step nine destroys accumulated reasoning state (github.com, 2023). A programmatic degradation ladder steps down capabilities across defined operational tiers — from Full Capability and Constrained Mode (RAG-only, no code execution) to Read-Only Mode, Human Takeover, and Hard Stop termination (substack.com, 2022; udemy.com, 2022).

A tactical executive blueprint installation detailing the four-phase implementation runbook for enterprise AI reliability engineering.

To protect execution graphs, Task-Decoupled Planning (TDP) separates supervisor planning nodes from individual execution nodes, enabling localized retries without abandoning safe checkpoints (github.com, 2023). Furthermore, mandate idempotency keys on all tool calls that enact state changes, neutralizing the risks of double-billing or duplicate resource provisioning during automated retry cycles (udemy.com, 2022).

VII. The Synthesis & CTO Runbook: Instrumenting OpenInference and Scaling the Autonomy Curve

Standardizing observability around OpenTelemetry and OpenInference semantic conventions allows organizations to capture end-to-end evaluation traces (Observe, Evaluate, Learn loops) without vendor lock-in (medium.com, 2024; arize.com, 2023). Deploying AI SRE agents requires scaling intentionally along the “Agentic Autonomy Curve,” moving safely from read-only triage to bounded production remediation (acceldata.io, 2024; substack.com, 2022; augmentcode.com, 2024).

CTO Phased Implementation Runbook:

• Phase 1 (Month 1): Instrument all LLM gateways using OpenInference standards to baseline Infrastructure and Behavioral SLIs.
• Phase 2 (Month 2): Deploy lightweight UQ companion models and establish baseline refusal ceilings and composite error budgets.
• Phase 3 (Month 3): Implement Quality-Aware Circuit Breakers and Model Bulkheads across high-risk agentic tool paths.
• Phase 4 (Month 4+): Activate phased degradation ladders and scale agentic autonomy along the governed incident-triage curve.

Download the enterprise-ready AIRe Open-Source Reference Architecture & Circuit Breaker Template on GitHub to audit your current prompt infrastructure and transition from probabilistic chaos to mathematically governed reliability engineering today.

References & Further Reading

Core Concepts

• Acceldata. (2024). Enterprise AI governance and hybrid architectures. Acceldata Research. https://www.acceldata.io
• Nintex. (2024). Deterministic vs. probabilistic workflows in modern business automation. Nintex Insights. https://www.nintex.com
• TestMu AI. (2024). Software delivery throughput and stability in the generative AI era. TestMu AI Research. https://www.testmuai.com
• Vinci Rufus. (2024). The historical evolution of deterministic computing. Vinci Rufus Tech Blog. https://www.vincirufus.com

Advanced Theory

• Massachusetts Institute of Technology. (2024). Uncertainty quantification in machine learning models. MIT Research Briefs. https://www.mit.edu
• ResearchGate. (2023). Adaptive workload modeling and resource allocation in cloud-native AI pipelines. ResearchGate. https://www.researchgate.net
• ResearchGate. (2024). Dynamic execution and probabilistic metrics for generative systems. ResearchGate. https://www.researchgate.net

Practical Applications

• Bigyan. (2024). Quality-aware circuit breakers for stochastic LLM applications. Bigyan Dev. https://www.bigyan.dev
• GetMaxim. (2024). Managing third-party LLM API failures and transient rate limits. GetMaxim AI. https://www.getmaxim.ai
• Microsoft. (2020). Cloud design patterns: Bulkhead and circuit breaker implementations. Microsoft Learn. https://www.microsoft.com
• Udemy. (2022). Defensive programming and idempotent design for distributed autonomous agents. Udemy Engineering. https://www.udemy.com

🌐 The GenAI & Quantum Intelligence Ecosystem

Explore our dedicated sister dispatches tailored to your specific focus lane:

• 🛡️ AI Safety & Governance: Tracking generative AI security, red teaming, and regulatory compliance? Subscribe to our flagship dispatch in Responsible Generative AI.
• 🔬 Deep Technical & arXiv: Want to inspect the formal proofs, benchmarks, and research architecture? Explore Generative AI Research.
• ⚛️ Quantum Security Bridge: Tracking quantum threats to AI security and cryptographic migration? Subscribe to Quantum.

🔔 Never miss a breakthrough: Subscribe to the publications above to build a multi-disciplinary vantage point across AI security, enterprise ROI, research benchmarks, and post-quantum encryption.

Disclaimer: The views and opinions expressed in this article are personal and do not necessarily reflect the official policy or position of any associated agencies, organizations, or the India AI Mission. AI assistance was utilized in the research, drafting, and ideation of this article. Licensed under CC BY-ND 4.0.

The Hidden Technical Debt of Prompt Hacks was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this article:

• Why a tool result marked success: true doesn't mean the agent did the right thing
• The three kinds of signal Claude Code and the Claude Agent SDK already expose: metrics, events, and traces
• Using PreToolUse and PostToolUse hooks to log tool calls without slowing the agent down
• How OpenTelemetry’s GenAI conventions turn a multi-step agent run into one readable trace
• What to instrument first if you’re starting from nothing

Why agent failures hide behind a green checkmark

A tool call that returns success: true tells you the function didn't throw an exception. It doesn't tell you the agent picked the right file, queried the right table, or interpreted the result correctly before acting on it. That gap between "the tool ran" and "the agent did the right thing" is where most production agent incidents actually live, and it's a distinction I've written about before in the context of validating model output rather than trusting confident-sounding responses at face value (Evaluating AI output: why it sounds right isn't good enough).

The problem compounds with every additional step in an agent loop. A single LLM call has one obvious failure mode: a bad response. A multi-step agent has a combinatorial one: the model can plan correctly but call the wrong tool, call the right tool with a malformed argument, get a correct result and misread it, or get a correct result and act on stale context from three turns earlier. None of those show up as an error in a basic request log, because nothing errored. Without step-level visibility, “why did the agent do that” becomes a question you answer by asking the model to explain itself after the fact, which is not a debugging method, it’s a guess with extra steps.

Anthropic’s own Claude Code exports exactly this kind of step-level data when you turn telemetry on, and walking through what it captures is a reasonable template for instrumenting any agent, not just Claude Code itself [1].

Three kinds of signal: metrics, events, and traces

Claude Code’s OpenTelemetry integration separates what it reports into three categories, and the distinction matters because each answers a different question [1]. Metrics answer “how much”: claude_code.token.usage breaks down tokens by type (input, output, cache read, cache creation), and claude_code.cost.usage reports session cost in USD, both tagged with attributes like model, query_source (main session, subagent, or auxiliary), and mcp_tool.name for per-tool cost attribution [1]. Events answer "what happened, when": the claude_code.tool_result event carries tool_name, duration_ms, success, and error_type, and every event in a single user turn shares a prompt.id so you can group them without stitching timestamps together yourself [1].

Traces answer the hardest question, “in what order, and how did the pieces relate.” Claude Code’s beta tracing support builds a span hierarchy with a root claude_code.interaction span containing child claude_code.llm_request and claude_code.tool spans, where each tool span further splits into claude_code.tool.blocked_on_user and claude_code.tool.execution [1]. That hierarchy is the actual answer to "was it the model or the tool": you can see the LLM request span's duration next to the tool execution span's duration in the same trace, rather than inferring it from two separate logs.

This three-layer split isn’t unique to Claude Code. It maps closely onto what OpenTelemetry’s own GenAI semantic conventions define for any LLM or agent system, which is worth knowing if you’re building on a different stack, and it’s the same layering that makes multi-agent systems traceable instead of opaque, something I ran into directly when writing about hub-and-spoke subagent orchestration in the Claude Agent SDK (Coordinator-subagent orchestration patterns in the Claude Agent SDK), where a coordinator’s trace and a subagent’s trace need to stay correlated without merging into one indistinguishable blob.

Logging tool calls with hooks, without slowing the agent down

Metrics and traces tell you what already happened. Hooks let you act on what’s about to happen, and they’re also the simplest way to add your own logging if you’re not ready to stand up a full OTel collector. The Claude Agent SDK fires a PreToolUse hook before a tool call executes and a PostToolUse hook after it returns, and each callback receives the tool name, its input arguments, and a tool_use_id that ties the two events to the same call [2]. A PreToolUse hook can log the call, block it, or rewrite its input before it runs; a PostToolUse hook can only observe and annotate, since the tool has already executed [2].

The matcher pattern on a hook determines which tools trigger it. An empty matcher runs the callback for every tool call, which is exactly what you want for a blanket audit log, while a matcher like Write|Edit scopes it to file-modifying tools, and MCP tools match the pattern mcp__<server>__<action>, which is worth knowing if your agent's failures cluster around one MCP integration rather than the built-in tools, a scenario I covered in more depth when the root cause is the tool description itself rather than the logging around it (MCP tool design: why your agent keeps calling the wrong tool) [2].

One detail matters for production use: if your hook’s only job is logging, don’t make the agent wait on it. The SDK supports an async output, {"async": true, "asyncTimeout": 30000}, that tells the agent to continue immediately while your callback finishes a webhook call or database write in the background [2]. A synchronous audit hook that calls out to a slow logging service will show up as latency in your claude_code.tool.execution span, which defeats the purpose of instrumenting for speed in the first place.

const auditLogger: HookCallback = async (input, toolUseID, { signal }) => {
  if (input.hook_event_name !== "PostToolUse") return {};

  // Fire-and-forget: don't block the agent on your logging pipeline
  sendToLoggingService({
    tool: input.tool_name,
    toolUseId: toolUseID,
    timestamp: new Date().toISOString(),
  }).catch(console.error);

  return { async: true, asyncTimeout: 30000 };
};

Turning a multi-step run into one trace

Claude Code’s span names are specific to Claude Code, but the underlying idea, that an agent run is a tree of spans rather than a flat log, is exactly what OpenTelemetry’s GenAI semantic conventions standardize for any agent framework. The conventions define a required gen_ai.operation.name attribute on every span, with values that map onto the actual shape of agent work: chat for a single inference call, execute_tool for a tool invocation, invoke_agent for a full agent run, invoke_workflow for multi-step orchestration, and plan for reasoning or task decomposition [4][5]. Recommended usage attributes like gen_ai.usage.input_tokens and gen_ai.usage.output_tokens sit on the chat spans, while gen_ai.agent.name and gen_ai.tool.name identify which agent or tool a given span belongs to, which is what makes it possible to filter a trace down to "just the subagent calls" or "just the retrieval steps" in a system with several moving parts [5].

OpenTelemetry’s own writeup of this frames the payoff in the same terms as the 45-second example at the start of this piece: a single trace with a root invoke_agent span containing child chat and execute_tool spans lets you see, in one view, where the time actually went, rather than reconstructing it from separate application logs after the fact [3]. That's a meaningfully different debugging experience from grepping through a session transcript, which is built to answer "what did the agent say," not "how long did each step take," a distinction worth keeping in mind if you're relying on transcript storage as your only record of what an agent did (The Three Layers of Memory in the Claude Agent SDK).

What to instrument first if you’re starting from nothing

You don’t need a collector, a dashboard, and a retention policy on day one. Start with whichever layer answers the question you get asked most often. If it’s “how much did that session cost,” turn on Claude Code’s metrics exporter and watch claude_code.token.usage and claude_code.cost.usage, both of which work with a plain console exporter before you touch OTLP [1]. If it's "did the agent do the thing it said it did," a single blanket PostToolUse hook with an empty matcher, logging tool name, input, and result to a flat file, gets you further than most people expect before you need a real observability platform [2].

If the question is “why did this specific run take so long, or go sideways three steps in,” that’s when tracing earns its setup cost, because a flat event log makes you reconstruct order and parent-child relationships by hand, and a trace hands them to you [1][3]. The honest failure mode to watch for either way is the same one that makes success: true misleading in the first place: a log that confirms a step executed without confirming it executed correctly. Tracing tells you where the agent spent its time. It doesn't tell you whether the agent was right, and no amount of span data replaces actually checking the output.

Sources

1. Monitoring — Claude Code Docs
2. Intercept and control agent behavior with hooks — Claude Code Docs
3. Inside the LLM Call: GenAI Observability with OpenTelemetry
4. OpenTelemetry Semantic Conventions for Generative AI (semantic-conventions-genai)
5. OpenTelemetry GenAI Semantic Conventions Explained — Dash0

If you enjoyed the article and wish to show your support, make sure to:

• 👏 Give a round of applause (50 claps) to help get featured
• 👥 Follow me on Medium to stay updated with my latest content
• 📰 Explore more articles on my Medium Profile
• 🔔 Connect with me on LinkedIn, Twitter, and Instagram

What to Log Before Your Agent Fails in Production was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.

I have a pet project: a companion app for Xteink X4, a deeply modified ESP32 with an e-ink display. Like most e-readers, it reads EPUB and nothing else. My books are PDFs — bought legally, in the only format the sellers ship — and my reader will not open a single one of them.

That is the bad half. The good half is that it pushed me into OCR, and into writing the thing this article is about: a PDF goes in, a structured document comes out, and an EPUB comes out of that. Everything below runs against PDFs I paid for or ones that are freely available.

Why not an LLM

This is the question I get asked the most, so it goes first. Two reasons.

First, I have not worked out how to run a fully local LLM inside a Kotlin Multiplatform project. Until that is solved, the whole thing stops being on-device, which was the point. Second, OCR is old, settled technology, and spending inference budget on work a mature recognizer already does well is a poor case for it. In a pipeline with a local model, OCR belongs in front of it: it prepares structured text cheaply instead of competing with the model for the same resources.

What OCR actually does

The pipeline has not changed much in thirty years, whatever engine you pick.

• Binarization — the page is reduced to ink and paper, because everything downstream operates on shapes rather than shades.
• Layout analysis — the page is split into blocks, blocks into lines, lines into words. This is where a two-column scan is either understood or ruined.
• Classification — individual glyphs, or whole lines, are turned into characters. Modern Tesseract does this with an LSTM over a whole line rather than character by character.
• Language model — a dictionary and character statistics correct what the classifier was unsure about. This is why feeding an engine the wrong language is worse than feeding it none.

Each platform ships its own implementation of that pipeline, and they are not the same implementation.

On iOS there is Vision, Apple’s own framework, available since iOS 13 and free with the OS. On Android there is Google’s ML Kit, and there is Tesseract through the Tesseract4Android binding. On a desktop JVM there is Tesseract again, through its C API.

So the choice made itself. Vision on iOS, because it is already there, costs nothing to ship, and uses hardware the phone already has. Tesseract on Android, because it is open, and because its language packs include various languages as a file I can bundle or download rather than a capability I have to hope for.

And the third target came free. Tesseract is the same engine on Android and on a desktop JVM, so supporting a JVM target cost one more binding rather than one more engine — which is also the only place I can run recognition end to end while developing, without a device in my hand.

What each engine gives you

Tesseract

Word and line boxes, a confidence per word, and language packs you control: .traineddata files you bundle, download at runtime, or point at an existing system installation. Through libtesseract's C API it also reports the script type of each line, the page orientation, and the skew angle. It needs to be told the real resolution of the image, through SetSourceResolution. That call must come after SetImage, or it is quietly ignored.

Vision

Word and line boxes, a confidence per observation, and two recognition levels, fast and accurate, that are a genuine speed-for-quality switch. It has no language packs to manage, because the languages are the ones the OS has. It does not report script type, orientation, or skew at all. Word boxes are not native either - they come from boundingBoxForRange, derived per range rather than reported directly.

The difference that matters is not in that list. It is that the same engine gives you different things depending on which binding you use: the Tesseract4Android Java wrapper does not surface the per-line script type, orientation and skew that the C API does. Two bindings of one library, two sets of capabilities.

Reducing three targets to one denominator

A cross-platform SDK has to answer this honestly, and there are only two honest answers: expose the intersection, or expose the difference. Uncial does both.

The intersection is what every target can do, and it is the default shape of the result: pages, lines, words, boxes, and a confidence on each.

The droppedAnyOf method matters more than it appears. An engine that cannot do a requested language drops it rather than failing, and asking is the only way to find out.

The difference is reported rather than smoothed over. It would be possible to return 0f for skew on every platform and call the API uniform. Uncial exposes UncialClient.capabilities instead, so the difference can be asked about. A uniform API that quietly returns a made-up value does not abstract the difference away - it just hides it.

Vision turns that from an inconvenience into a trap. Its set of supported languages is a property of the request, not of the framework — the documentation for supportedRecognitionLanguages says a language available at one recognition level may be absent at another. There is no Ukrainian at all before iOS 16.

The document, not the string

Text recognition takes an image and returns a string. Abstract, but workable. It is usually enough, too: a receipt, a meter reading, anything that goes straight into an input field. But the moment you need to highlight a matched word on the scan itself, the position is not in that string, and there is nowhere left to get it from.

• Position — every word’s box on the page. Without it you cannot highlight a search hit, crop a field out of a form, or draw anything at all over the scan.
• Confidence — how sure the engine is of each word. A string does not carry that at all.
• Reading order — in a two-column scan, reading straight down mixes lines from both columns together.
• Type size — the one signal that separates a heading from a paragraph, and it exists only upstream of the string.

That last one is what makes an EPUB possible at the other end. Without type size there are no headings, and without headings there is no table of contents — only one long undifferentiated wall of text.

Here is exactly what comes back. That is the whole contract:

document.pages[0].lines[0] is an OcrLine: its text, its box in page pixels, its fontSize, a Confidence, and the OcrWord list underneath it. blocks is the same content in reading order as Heading(level) and Paragraph.

I am an Android developer, not an OCR researcher. The structure heuristics — the ratio of a line’s type size to the document’s median that promotes it to a heading, the repeat frequency that marks a running header as page furniture — were tuned by hand and live in StructureOptions precisely because they are document-dependent. A technical manual with 11 pt body and 12 pt headings needs different numbers than a novel does.

One of those heuristics has a trap under it worth naming. PDFBox reports a font size of 0 pt for Type3 fonts and degenerate text matrices, so a real document can mix genuine sizes with zeros. Let the zeros vote in the median and the median sinks below the body size, at which point ordinary prose gets promoted to headings. NaN is the same trap from the other side, since every comparison against it is false: a line with an unreported size would sail past the ratio test and become a heading.

The fix is a filter ahead of the median — only finite sizes above zero get a vote.

The case where nothing survives that filter is handled separately: a document from a recognizer that reports no type sizes at all yields a median of 0f, and then no line is promoted and everything becomes a Paragraph. That is deliberately better than returning an empty list, because the text is there - there is simply no basis for calling any of it a heading.

One page, three coordinate systems, pick one

Vision reports normalized coordinates with the origin at the bottom-left. PDF is bottom-up too, but in points rather than pixels. PdfRenderer and PdfBox give you top-down pixels.

If the SDK handed back each platform’s coordinates unchanged, the conversion would land in the app — with different arithmetic for Android, iOS and the JVM.

Uncial picks one model — top-down pixels, origin at the page’s top-left — and each engine converts at its own edge. Here is that conversion in Vision, written once:

Three lines of arithmetic that would otherwise live in every consuming app. Nobody downstream ever branches on the platform to read a box. The digital text path does the same thing from the other direction: pdf-text scales PDF points by the render scale, so a page that came out of the embedded text layer and a page that came out of OCR are directly comparable inside one document.

Two constants govern how a page becomes pixels in the first place, and both are load-bearing.

renderDpi defaults to 200. Below roughly 150 dpi Tesseract's accuracy on printed text falls off a cliff, because the LSTM was trained on glyphs of a certain pixel height and small type stops clearing it. Above roughly 300 you pay memory and time for detail the recognizer discards during binarization anyway. 200 is the flat part of that curve for printed documents, which is what this SDK is for.

maxPageSide caps the longer side of a rasterized page at 2600 px and is applied after the dpi, scaling down when it has to. This is purely an out-of-memory (OOM) guard: an A0 poster at 200 dpi is over 9000 px across, and a grayscale buffer that size is well past what a cheap phone will hand you in one allocation. Grayscale is the default pixel format for the same reason - engines binarize regardless, and one channel is worth a quarter of ARGB, which is the difference between a 300-page scan finishing and dying.

The consequence is a trap I paid for twice: the dpi you must give Tesseract is not renderDpi - it is the effective dpi after the maxPageSide clamp. Raster.sourceDpi carries that real figure, and it is deliberately left null for an image the caller supplied, because in that case nobody rendered anything and the honest move is to say nothing and let Tesseract estimate.

The runtime module does not know that engines exist

runtime has no compile-time dependency on any recognizer. Engines announce themselves through a registry in core: on Android the Tesseract engine registers itself via androidx.startup, and everywhere else the consumer calls installTesseractEngine() or installVisionEngine() once.

The reason is practical. Tesseract on Android means a native library plus language data measured in megabytes, and somebody who only ever OCRs camera photographs on iOS should not link any of it. The same rule is why the PDF text-layer reader is a separate module that you opt into by setting digitalTextExtractor = pdfTextExtractor() - on Android it pulls in several MB of PDFBox-Android, which is a real cost to force on a consumer who never opens a PDF.

Rasterizing and recognizing are also separate roles, PageRasterizer and TextRecognizer. That split is what makes it possible to feed Uncial a PNG instead of a PDF, to test the whole pipeline against a fake recognizer with no native code in sight, and to swap Tesseract for Vision without the calling code noticing.

This is also the place to say why KMP rather than a Rust core with thin platform wrappers, which is the other common shape for an SDK like this. It buys nothing here: Vision belongs to Apple and will not link into a Rust core, so you would still be writing two native bindings, and you would have added a third language to the build.

There is no DI framework inside the SDK, on purpose. A DI dependency in a library is a version conflict you inflict on every consumer, so the composition root is a hand-written builder.

One codebase, two completely different deliveries

An Android consumer can assemble a set of Gradle modules. An iOS consumer gets one binary. That asymmetry produces two artifact shapes with almost nothing in common, over one codebase.

iOS is an umbrella. A separate module builds a static framework and re-exports everything that appears in the public API:

export is not cosmetic there. Without it Swift gets types that exist but cannot be named, and the whole document model becomes unreachable. This is also the one place where the rule that runtime knows nothing about engines stops applying: a distribution artifact is exactly where they are supposed to meet.

A thin UncialIos wrapper sits next to it and covers the three places where the generated Objective-C API gets awkward: ByteArray, the builder lambda, and Result, which does not survive the export at all.

Android is a fat AAR. The reason is not convenience; it is that the alternative does not work. Tesseract4Android lives on JitPack, and as a plain api dependency it would force every consumer to add the JitPack repository to their own settings.gradle.kts or watch resolution fail. That is unacceptable in an SDK. On top of that: nine artifacts for one SDK, and native code and assets that have to arrive together with the classes, or TessBaseAPI.init() fails at runtime with an unhelpful error.

The first release ships the twelve modules on their own; the fat AAR is the step after it, and until then an Android consumer still adds the JitPack repository for Tesseract4Android — which is the very thing it exists to remove.

The rule for splitting is simple. Embed what is our implementation detail and what the consumer will never name in their own code. Keep external what they almost certainly already have, or what needs the build system’s participation.

Two consequences are easy to miss. Native code reaches Java through JNI reflection, which R8 cannot see, so consumer ProGuard rules are mandatory — without them the release builds and then crashes. And if the jar merge loses META-INF/*.kotlin_module, the consumer quietly loses default arguments, suspend breaks, and top-level functions disappear. That is not a hypothetical, it is the standard way to get a fat AAR wrong.

This is why the acceptance criterion is not just “it built”. A clean new Android project, google() and mavenCentral() only, one dependency line, R8 enabled, a release build - and OCR of a Ukrainian scan returns text. Until that passes, the fat AAR is not done.

Where it lives. The modules publish to Maven Central under the io.github.andrew-malitchuk group, the namespace verified through the GitHub account itself — free and immediate. Publication goes through com.vanniktech.maven.publish, because Kotlin Multiplatform turns 12 Gradle modules into 58 Maven coordinates — a root module per library plus one per target — and nobody should hand-write that many publications. Its own uploader, though, could not be used. It reads credentials only through providers.gradleProperty, and a project-local secrets file can never become one: Gradle 9 loads its properties before the settings script runs, so injecting them from settings.gradle.kts silently does nothing. The upload therefore goes through nmcp, which takes credentials as ordinary values and aggregates all 58 coordinates into a single Portal deployment instead of twelve.

iOS travels a separate road, and SPM dictates the order of it. The manifest is read at the git tag, and the checksum pins the exact bytes of the zip — so you commit a URL to a GitHub Release asset that does not exist yet, then tag, then upload. That looks backwards and is the only order that works. Package.swift lives in this repository rather than in one of its own, so a single tag carries both the Maven release and the Swift one and no second manifest can drift away from the artifact it names.

• https://github.com/andrew-malitchuk/uncial-kmp
• https://central.sonatype.com/artifact/io.github.andrew-malitchuk/uncial-core/0.0.1

The one part already acting as a release gate is the ABI dump. Every published module has strict explicitApi() and a committed dump, and checkKotlinAbi runs as part of build. A change to the public API fails the build until somebody regenerates the dump and reads the diff.

Green tests that prove nothing

Vision text recognition returns zero observations on the iOS Simulator. There is no Neural Engine there, so performRequests succeeds, error is null, and results is an empty array. Every assertion I had written passed. The pipeline had recognized precisely nothing.

That produced the rule the tests are written to now, and it generalizes to any KMP library over a native dependency: an assertion may only check what the environment is actually able to perform. The iOS tests assert the pipeline and the capabilities, and check text only when Vision actually returned some. A green iOS run says the pipeline is assembled correctly. It says nothing about recognition quality, and only a physical device can. In code:

The out-of-page check is not cosmetic. If the coordinate conversion above ever breaks, boxes land outside the page, and this is the only assertion that catches it.

There are more places like it:

• org.gradle.caching=true means a green build can be entirely cache hits with no test actually executed, which is what --rerun-tasks is for.
• Android host tests cannot construct a Bitmap, so the fake rasterizer produces a placeholder that carries dimensions and no pixels.
• Unannotated Objective-C and JNA signatures type as non-null in Kotlin while genuinely returning null, so the compiler will call your ?: throw dead code right up until it fires.

The useful artifact that came out of all this is not a test suite. It is a written list of what the test suite does not check, kept in the repository next to the code.

What this was actually worth

Not wanting to buy an e-ink Android reader, and not wanting to read on a tablet, is the entire reason a KMP OCR library exists on my disk.

There is no claim of novelty here. I did not invent a recognition algorithm, alone or with a model’s help — I am not competent to, and there are solutions on the market that do the recognition part far better than anything I would produce. Everything above is assembly, not research.

The real value of this experience lies in the second half. This is a clean, almost textbook case of building a cross-platform SDK over a problem where every platform already has its own answer and no two answers match. You end up building a facade that has to deliver something coherent anyway: trade-offs, palliative decisions, and a lot of knowledge that only arrives by running the thing. As an exercise in what a KMP SDK costs to build, it was worth every hour.

And it closed the need it started from. A PDF I own goes in, and an EPUB my reader will open comes out.

Source:

• https://github.com/andrew-malitchuk/uncial-kmp
• https://github.com/andrew-malitchuk/inkcast-kmp

GitHub | LinkedIn | X

OCR as a Document, Not Just a String was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.