ai-toolkit / LLM building blocks

Engineering LLM Applications that Fail Predictably: A Toolkit for Ingestion, Retrieval, Evaluation and Routing

Thirteen Python packages for ingestion, retrieval, evaluation and routing that share one model gateway, so failures surface, spending stays bounded, logs stay private and measurements can be trusted.

13packages in one uv workspace, each installable on its own
1 → 3one gateway for Gemini, OpenAI and local Ollama models
4CI gates: tests, ruff, mypy and pip-audit

Overview—Most LLM tooling is written for the happy path, yet production systems meet rate limits, provider outages, scanned documents, interrupted jobs, confidential data and unbounded bills. This toolkit covers document ingestion, retrieval, evaluation, query routing and agents, and routes every model call through one gateway that owns failures, cost, privacy and tracing. Failures are explicit rather than silently absorbed, spending is bounded, logs are private by default, and evaluation results come with confidence intervals and survive crashes.

IIntroduction

An LLM application is a distributed system built around an unreliable, metered and non-deterministic dependency. Common defaults treat that dependency as an ordinary function call: retry anything, switch models when one fails, average whatever scores come back. Each default is reasonable on its own. Together they produce systems that look healthy while answering from the wrong model, losing data and overstating improvements.

This project takes the opposite position: a failure should be visible, bounded and attributable. Rather than a framework, it is a set of small packages with explicit contracts that share one gateway, so the rules for failure, cost and privacy are written once and apply everywhere.

IIArchitecture

The packages form three stages (Fig. 1). Build turns documents into a searchable index, Serve routes and answers user queries, and Measure generates test questions, benchmarks retrieval and scores answers. No component calls a model provider directly; every call goes through the gateway.

Architecture: build, serve and measure stages, with every model call routed through llm-gateway to Gemini, OpenAI or Ollama Build Serve Measure Documents PDF · DOCX · scans doc-ingest chunks · OCR retrieval-kit BM25 · dense · hybrid RAG app agent-loop · MCP router rules · embed · LLM User query any language synth-data questions · labels benchmark recall · MRR · nDCG eval-harness judges · CIs · resume llm-gateway timeouts · retries · rate limits · spending caps · caches · cost logs · tracing Providers Gemini · OpenAI · Ollama index chunks index answers test questions OCR · embeddings generation · judges routing · answers
Fig. 1. Three stages share one gateway. Solid arrows carry data; dashed arrows are model calls.
calls a model (through the gateway) deterministic code gateway: the single path to providers

Table IPackages

PackageResponsibilityStatus
llm-gatewayOne client for Gemini, OpenAI and Ollama, with failure handling, limits, spending caps, caching, cost accounting and tracingcore
doc-ingestPDF, DOCX, HTML and scans into chunks with page, section and OCR metadatacore
retrieval-kitLexical, dense and hybrid retrieval, rerankers and benchmarkscore
eval-harnessRule-based metrics and LLM judges, run comparisons with confidence intervalscore
guardrails-routerQuery routing through rules, then embeddings, then an LLMcore
eight moreTest-set generation, prompt versioning, schema extraction, a tool-calling agent, MCP access, a model playground, corpus maps, speechexperimental

IIIFailure Semantics

The gateway decides what happens when a call fails, once and the same way for every package (Fig. 2). Only transient failures (throttling, server errors, timeouts, dropped connections) are retried or passed to a fallback model; a bad request or an authentication error surfaces immediately, because answering from another model would hide it. A reply cut off at the token limit is reported as truncated rather than treated as an answer. In ingestion, a page that cannot be read is skipped on its own instead of taking its document with it. The query router sends a query it cannot classify to a cautious route, such as asking for clarification, never to the normal answer path.

  1. Cache lookupAn identical earlier request is answered from the cache at no cost.
    free
  2. Spending checkThe call is refused if the client's or the day's budget is spent.
    caps
  3. Provider slotThe call waits for the provider's shared rate and concurrency budget.
    limits
  4. Call with a timeoutNo call can hang indefinitely.
    always
  5. Retry transient failuresThrottling, server errors and dropped connections, honouring the provider's requested delay.
    backoff
  6. Fall back to another modelOnly for transient failures; other errors surface.
    optional
  7. Account and recordCost from token counts, a metadata-only log entry and a trace span.
    always
Fig. 2. The path of one model call through the gateway.

IVCost and Concurrency

Every concurrent job in a process (evaluation, OCR, batch generation) draws from one shared rate and concurrency budget per provider, instead of each backing off on its own after being throttled. Spending can be capped per client and per day, and the daily cap holds across separate processes because it is computed from the shared call log. Work that has been paid for once is not paid for again: embeddings and OCR results are cached by content, so rebuilding an index or re-ingesting a document reuses earlier results (Table II).

VMeasurement

An evaluation tool that misleads is worse than none, so each run is built to be reproducible and comparable. Examples are identified by their content rather than their position, so editing a dataset never mismatches results. Two runs are compared with paired bootstrap confidence intervals [3], and a change is reported as significant only when its interval excludes zero. Results are saved as they are produced, so an interrupted run resumes where it stopped, and every run records the models, data and code version that produced it. Retrieval labels are stored as positions in the source document rather than chunk identifiers, so different chunking strategies can be compared against the same labels.

VIModel Independence

Behaviour should not change silently when a different model is configured. Similarity scores differ between embedding models, so a fixed routing threshold would route the same query differently under each one; the router therefore calibrates its threshold from its own examples. PDFs carry no section markup, so the ingester recovers section headings from typography (font size and weight), giving chunks from PDFs the same section context as those from DOCX or HTML.

VIIPrivacy and Security

Private by default

Logs record metadata such as model, tokens, cost and latency. Prompts and replies are kept only when explicitly enabled, and caches store no document text.

Untrusted documents

Retrieved passages are fenced off in prompts and marked as reference material, limiting indirect prompt injection [4].

Authenticated services

Network-facing MCP servers [6] require a token and refuse to serve beyond the local machine without one.

Contained configuration

Settings are read only from within the project, and ingestion limits file sizes and page counts.

VIIIEngineering Practice

The test suite runs offline against a simulated provider, and contract tests check each provider adapter against the SDK's own response types. Continuous integration runs the tests on Linux and Windows together with linting, type checking and a dependency vulnerability audit, and a daily job makes real provider calls. Every model call emits an OpenTelemetry span [5] with its tokens, cost, retries and errors. Releases follow semantic versioning.

Table IIMeasured effects of engineering changes

ChangeBeforeAfterMeasured on
Embedding cache58 s1 srebuilding a 2,156-chunk index
OCR cache41 s3 sre-ingesting a scanned book, identical output
Vectorised filters73 ms27 msfiltered search over 200,000 chunks
PDF section headings171,840chunks with a section path, of about 2,160
Page-level failure isolationdocument lostone page skippeda provider error on one page of 279
Calibrated routing threshold0.80 fixed0.59 / 0.66two embedding models (nomic, Gemini)

IXLimitations

  • Answer quality has not yet been measured on a human-reviewed test set; the results above concern engineering behaviour.
  • Section-heading recovery is heuristic and may need tuning for unusual PDF layouts.
  • Dense retrieval runs in memory, which suits corpora up to a few hundred thousand chunks.

XImplementation

About 7,000 lines of Python in thirteen packages, released under the MIT licence.

Language and buildPython 3.11+, uv workspace [7]
ModelsGoogle GenAI and OpenAI SDKs, Ollama, Pydantic
RetrievalBM25 [1], dense search, reciprocal rank fusion [2]
IngestionPyMuPDF, pypdf, python-docx, BeautifulSoup
EvaluationSQLite run store, LLM judges, bootstrap statistics
Qualitypytest, ruff, mypy, pip-audit, GitHub Actions

References

  1. S. Robertson and H. Zaragoza, “The probabilistic relevance framework: BM25 and beyond,” Foundations and Trends in Information Retrieval, vol. 3, no. 4, pp. 333–389, 2009.
  2. G. V. Cormack, C. L. A. Clarke, and S. Büttcher, “Reciprocal rank fusion outperforms Condorcet and individual rank learning methods,” in Proc. 32nd ACM SIGIR Conference, 2009, pp. 758–759.
  3. B. Efron and R. J. Tibshirani, An Introduction to the Bootstrap. New York: Chapman & Hall, 1993.
  4. K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection,” in Proc. 16th ACM Workshop on Artificial Intelligence and Security (AISec), 2023.
  5. OpenTelemetry, “Semantic conventions for generative AI systems.” [Online]. Available: opentelemetry.io/docs/specs/semconv/gen-ai
  6. “Model Context Protocol specification.” [Online]. Available: modelcontextprotocol.io
  7. Astral, “uv: An extremely fast Python package and project manager.” [Online]. Available: github.com/astral-sh/uv