Overview—Most LLM tooling is written for the happy path, yet production systems meet rate limits, provider outages, scanned documents, interrupted jobs, confidential data and unbounded bills. This toolkit covers document ingestion, retrieval, evaluation, query routing and agents, and routes every model call through one gateway that owns failures, cost, privacy and tracing. Failures are explicit rather than silently absorbed, spending is bounded, logs are private by default, and evaluation results come with confidence intervals and survive crashes.
IIntroduction
An LLM application is a distributed system built around an unreliable, metered and non-deterministic dependency. Common defaults treat that dependency as an ordinary function call: retry anything, switch models when one fails, average whatever scores come back. Each default is reasonable on its own. Together they produce systems that look healthy while answering from the wrong model, losing data and overstating improvements.
This project takes the opposite position: a failure should be visible, bounded and attributable. Rather than a framework, it is a set of small packages with explicit contracts that share one gateway, so the rules for failure, cost and privacy are written once and apply everywhere.
IIArchitecture
The packages form three stages (Fig. 1). Build turns documents into a searchable index, Serve routes and answers user queries, and Measure generates test questions, benchmarks retrieval and scores answers. No component calls a model provider directly; every call goes through the gateway.
Table IPackages
| Package | Responsibility | Status |
|---|---|---|
| llm-gateway | One client for Gemini, OpenAI and Ollama, with failure handling, limits, spending caps, caching, cost accounting and tracing | core |
| doc-ingest | PDF, DOCX, HTML and scans into chunks with page, section and OCR metadata | core |
| retrieval-kit | Lexical, dense and hybrid retrieval, rerankers and benchmarks | core |
| eval-harness | Rule-based metrics and LLM judges, run comparisons with confidence intervals | core |
| guardrails-router | Query routing through rules, then embeddings, then an LLM | core |
| eight more | Test-set generation, prompt versioning, schema extraction, a tool-calling agent, MCP access, a model playground, corpus maps, speech | experimental |
IIIFailure Semantics
The gateway decides what happens when a call fails, once and the same way for every package (Fig. 2). Only transient failures (throttling, server errors, timeouts, dropped connections) are retried or passed to a fallback model; a bad request or an authentication error surfaces immediately, because answering from another model would hide it. A reply cut off at the token limit is reported as truncated rather than treated as an answer. In ingestion, a page that cannot be read is skipped on its own instead of taking its document with it. The query router sends a query it cannot classify to a cautious route, such as asking for clarification, never to the normal answer path.
- Cache lookupAn identical earlier request is answered from the cache at no cost.free
- Spending checkThe call is refused if the client's or the day's budget is spent.caps
- Provider slotThe call waits for the provider's shared rate and concurrency budget.limits
- Call with a timeoutNo call can hang indefinitely.always
- Retry transient failuresThrottling, server errors and dropped connections, honouring the provider's requested delay.backoff
- Fall back to another modelOnly for transient failures; other errors surface.optional
- Account and recordCost from token counts, a metadata-only log entry and a trace span.always
IVCost and Concurrency
Every concurrent job in a process (evaluation, OCR, batch generation) draws from one shared rate and concurrency budget per provider, instead of each backing off on its own after being throttled. Spending can be capped per client and per day, and the daily cap holds across separate processes because it is computed from the shared call log. Work that has been paid for once is not paid for again: embeddings and OCR results are cached by content, so rebuilding an index or re-ingesting a document reuses earlier results (Table II).
VMeasurement
An evaluation tool that misleads is worse than none, so each run is built to be reproducible and comparable. Examples are identified by their content rather than their position, so editing a dataset never mismatches results. Two runs are compared with paired bootstrap confidence intervals [3], and a change is reported as significant only when its interval excludes zero. Results are saved as they are produced, so an interrupted run resumes where it stopped, and every run records the models, data and code version that produced it. Retrieval labels are stored as positions in the source document rather than chunk identifiers, so different chunking strategies can be compared against the same labels.
VIModel Independence
Behaviour should not change silently when a different model is configured. Similarity scores differ between embedding models, so a fixed routing threshold would route the same query differently under each one; the router therefore calibrates its threshold from its own examples. PDFs carry no section markup, so the ingester recovers section headings from typography (font size and weight), giving chunks from PDFs the same section context as those from DOCX or HTML.
VIIPrivacy and Security
Private by default
Logs record metadata such as model, tokens, cost and latency. Prompts and replies are kept only when explicitly enabled, and caches store no document text.
Untrusted documents
Retrieved passages are fenced off in prompts and marked as reference material, limiting indirect prompt injection [4].
Authenticated services
Network-facing MCP servers [6] require a token and refuse to serve beyond the local machine without one.
Contained configuration
Settings are read only from within the project, and ingestion limits file sizes and page counts.
VIIIEngineering Practice
The test suite runs offline against a simulated provider, and contract tests check each provider adapter against the SDK's own response types. Continuous integration runs the tests on Linux and Windows together with linting, type checking and a dependency vulnerability audit, and a daily job makes real provider calls. Every model call emits an OpenTelemetry span [5] with its tokens, cost, retries and errors. Releases follow semantic versioning.
Table IIMeasured effects of engineering changes
| Change | Before | After | Measured on |
|---|---|---|---|
| Embedding cache | 58 s | 1 s | rebuilding a 2,156-chunk index |
| OCR cache | 41 s | 3 s | re-ingesting a scanned book, identical output |
| Vectorised filters | 73 ms | 27 ms | filtered search over 200,000 chunks |
| PDF section headings | 17 | 1,840 | chunks with a section path, of about 2,160 |
| Page-level failure isolation | document lost | one page skipped | a provider error on one page of 279 |
| Calibrated routing threshold | 0.80 fixed | 0.59 / 0.66 | two embedding models (nomic, Gemini) |
IXLimitations
- Answer quality has not yet been measured on a human-reviewed test set; the results above concern engineering behaviour.
- Section-heading recovery is heuristic and may need tuning for unusual PDF layouts.
- Dense retrieval runs in memory, which suits corpora up to a few hundred thousand chunks.
XImplementation
About 7,000 lines of Python in thirteen packages, released under the MIT licence.
References
- S. Robertson and H. Zaragoza, “The probabilistic relevance framework: BM25 and beyond,” Foundations and Trends in Information Retrieval, vol. 3, no. 4, pp. 333–389, 2009.
- G. V. Cormack, C. L. A. Clarke, and S. Büttcher, “Reciprocal rank fusion outperforms Condorcet and individual rank learning methods,” in Proc. 32nd ACM SIGIR Conference, 2009, pp. 758–759.
- B. Efron and R. J. Tibshirani, An Introduction to the Bootstrap. New York: Chapman & Hall, 1993.
- K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection,” in Proc. 16th ACM Workshop on Artificial Intelligence and Security (AISec), 2023.
- OpenTelemetry, “Semantic conventions for generative AI systems.” [Online]. Available: opentelemetry.io/docs/specs/semconv/gen-ai
- “Model Context Protocol specification.” [Online]. Available: modelcontextprotocol.io
- Astral, “uv: An extremely fast Python package and project manager.” [Online]. Available: github.com/astral-sh/uv