AI infrastructure / production guide

Production LLM and RAG readiness checklist

A practical checklist for evaluation, retrieval quality, reliability, observability, latency, cost, security boundaries, and operational ownership.

Book an assessment-fit call ↗

By: Francisco Perez Romero · 2026-08-07

Decision guide

Evidence before prescription.

Define quality before optimizing it

A production AI system needs representative scenarios and explicit failure categories. Generic benchmark scores and impressive demos do not show whether the workflow succeeds for real users.

Representative scenarios drawn from authorized production patterns.
Success, partial-success, abstention, and harmful-failure criteria.
Evaluation ownership, cadence, and regression thresholds.
Separate retrieval, generation, tool, and workflow failures.

Trace the whole system

RAG and agent systems are distributed systems. Quality and latency emerge from data sources, indexing, retrieval, reranking, prompts, models, tools, fallbacks, and human review—not from the model alone.

Trace identifiers across retrieval, generation, tools, and fallbacks.
Latency, token, model, vector, and infrastructure cost per outcome.
Freshness and access-control behavior at the data boundary.
Capacity, timeout, retry, degradation, and provider-outage behavior.

Bound authority and failure

The system must know when not to answer or act. Permissions, prompt-injection controls, sensitive-data boundaries, human review, and auditable tool execution should match the consequence of the workflow.

Least-privilege tool and data access.
Prompt-injection and untrusted-content handling.
Fallback and escalation paths for uncertain results.
Rollback, incident response, and evaluation after change.

Frequently asked

How do you know whether a RAG system is production ready?

It has representative evaluation, observable retrieval and generation stages, defined failure behavior, bounded data and tool access, operational owners, and acceptable quality, latency, reliability, and cost under realistic load.

Should we change models before improving retrieval?

Not automatically. Trace failures first. Poor answers may come from source quality, chunking, retrieval, ranking, prompting, model behavior, tool execution, or missing abstention. The evidence should determine the intervention.

Continue the decision

AI infrastructure consulting

Scope evaluation, reliability, observability, performance, and production hardening.

Open →

Infrastructure operating proof

Review Francisco's attributed experience at global scale.

Open →