50 Checks

LLM Production Readiness

A pragmatic checklist to ship LLM systems safely: architecture, data & safety, evals, operations & cost.

Completed

Progress

Total

Architecture0/7

Infrastructure, security, and reliability foundations

Defined SLOs for latency (p50/p95) and availability

VPC/VNet or on-prem isolation for sensitive data

Secret management (KMS/Vault), key rotation policy

Retry/backoff, circuit-breakers, and fallbacks

Request/response size limits; streaming enabled

Versioned prompts, configs, and model routing

Multi-region failover plan documented

Data & Safety0/6

Privacy, content policy, and safety controls

PII/PHI detection and redaction at ingest

Content policy & guardrails enforced (allow/deny)

Toxicity and jailbreak filters configured

Document source access controls and tenancy

Dataset lineage & consent recorded

Audit trail for content updates and takedowns

Retrieval & RAG0/7

Indexing, search, and context assembly

Chunking strategy validated on representative docs

Embedding model chosen; recall@k measured

Vector DB schema & filters tested

Hybrid search (dense+sparse) evaluated

Re-ranking improves precision on eval set

Context assembly enforces token budgets

Citations/anchors included in responses

Generation & Policies0/5

Response generation and output controls

Response templates or JSON schemas for structure

Explicit refusal rules ("not found" without evidence)

Style/voice controls documented

Guarded tool-use (CRM/email/actions) with scopes

Rate limits per user/tenant

Evals & QA0/5

Testing, validation, and quality assurance

Golden set with exact answers & references

Faithfulness/grounding evals automated

Red-team scenarios run regularly

Regression gates block risky deploys

Human-in-the-loop review workflow (as needed)

Observability & Ops0/5

Monitoring, alerting, and operations

Prompt/context/output tracing with PII scrubbing

Latency, error, refusal, and cost dashboards

Drift detection on retrieval quality and outputs

Alerting on SLO breaches and anomalies

On-call runbook & rollback plan documented

Cost & Performance0/5

Optimization, budgeting, and efficiency

Quantization strategy tested (INT8/FP8/etc.)

Batching opportunities identified

Cache hit-rate monitored and tuned

Cost per request/answer tracked with budgets

Vendor pricing reviewed; alternative routes ready

Extras0/10

Advanced practices and preparation