AI Six Sigma · Continuous monitoring
Statistical process control for model behaviour.
We run AI systems the way process engineers run a production line: every decision is a unit, every critical-to-quality characteristic has a written defect definition, and a control chart — not a glance at a dashboard — decides when behaviour has shifted and who gets paged.
Method
DMAIC, applied to AI systems.
Define, Measure, Analyze, Improve, Control is the Six Sigma improvement cycle. It transfers to AI with one change: the unit of production is a decision, and the process includes the prompt, the model, the retrieval index and every tool the model calls.
Define what a defective decision is — before anyone looks at a dashboard.
Each AI decision is a unit of output. We agree the critical-to-quality characteristics with the business and risk owners, then write each one as a testable defect definition. A decision that violates any CTQ is defective, which makes every decision a Bernoulli trial and the daily defect proportion a statistic we can put limits on.
| CTQ | Defect definition | Measurement | Example tolerance |
|---|---|---|---|
| Groundedness | Any claim in the output that is not entailed by the retrieved context, judged by an NLI entailment model calibrated against a human-labelled golden set. | Claim-level entailment on every sampled decision | ≤ 0.5% of decisions with an unsupported claim |
| Policy adherence | A deterministic policy check or guardrail classifier fails on the final output — PII egress, prohibited advice, jurisdiction rules — or a required disclosure is missing. | Rule engine + classifier verdicts, logged per decision | 0 critical; ≤ 0.1% minor |
| Latency p95 | End-to-end decision latency above the SLO, measured from the gateway span to the committed response. | Per-request trace span; EWMA chart on the daily p95 | ≤ 1,800 ms per decision |
| Cost per decision | Metered tokens, tool calls and retrieval for one decision exceed the budget ceiling, joined on trace id. | Usage records joined to the decision ledger | ≤ $0.042 per decision |
| Escalation precision | The system escalates and the reviewer marks it unnecessary, or it does not escalate a case the reviewer later reverses. | Reviewer labels on escalations + 5% audit sample | Precision ≥ 0.90, recall ≥ 0.97 |
Tolerances are illustrative; we set them per system with the accountable owner and record them in the control plan.
Measure with instruments that are themselves measured.
An LLM judge is a gauge, and gauges need a repeatability and reproducibility study. We validate every automated judge against human labels before its output feeds a chart, then sample production daily so the chart reflects real traffic rather than a benchmark.
- Eval suites
- One check per CTQ, run in CI on every change and nightly against sampled production. Deterministic checks first (schema, policy rules, citations), model judges second.
- Golden sets
- Stratified by segment and difficulty, versioned alongside the prompt and model they gate, with a sealed holdout. Judge agreement with humans is tracked as Cohen's κ, target ≥ 0.8.
- Drift metrics
- Population Stability Index on intent mix and embedding-cluster assignment (0.1 watch, 0.2 act); KL divergence on output label distributions against a rolling reference window.
- Latency and cost SPC
- Daily p95 latency and cost per decision on EWMA charts (λ = 0.2), sensitive to the small sustained shifts that follow a provider update or a prompt that quietly grew by 900 tokens.
Analyze failures by mode, not by anecdote.
Every defect is classified against a failure-mode taxonomy, then Pareto-ranked by segment. Because every prompt, model, index and tool change is a versioned event in the decision ledger, a shift on the chart can be aligned to the change that preceded it and the affected decisions replayed exactly.
| Failure mode | Detection signal | Typical root cause |
|---|---|---|
| Ungrounded claim | NLI entailment failure; cited span does not contain the asserted fact | Retrieval recall dropped after an index rebuild or chunking change; context truncated by a longer system prompt |
| Retrieval miss | Golden-set recall@k falls; top-k similarity scores shift down | Embedding model version changed without re-indexing; stale index; query-rewrite prompt regression |
| Policy breach | Guardrail classifier hit or deterministic rule failure on the final output | Prompt edit removed a constraint; new input segment outside the tested distribution; adversarial pattern |
| Schema violation | JSON-schema validation failure rate on structured outputs | Provider model update changed output tendencies; constrained decoding not enforced |
| Tool-call error | Tool error rate and retries per decision | Upstream API contract change; model invents arguments the schema does not constrain |
| Latency blowout | p95 span duration breaches EWMA limits; queue depth rises | Provider throttling; context growth; router shifting traffic to the large tier |
| Cost overrun | Tokens and tool calls per decision trend up on the cost chart | Prompt growth; retry loops; semantic-cache hit rate collapse after a key change |
| Over-escalation | Escalation precision falls against reviewer labels | Confidence scores not recalibrated after a model swap; threshold set on a stale validation set |
| Input drift | PSI > 0.2 on intent mix or embedding-cluster assignment; KL on output labels | New product, channel or season changes who is asking what; upstream form change |
Improve with the smallest intervention that moves the CTQ.
We fix at the cheapest layer that addresses the root cause, and we prove the fix with a pre-registered test: the CTQ, the minimum detectable effect and the sample size are written down before the experiment starts.
- Prompt interventions
- Constraint rewrites and few-shot examples drawn from the failure cluster, diffed and versioned; each change re-runs the full golden set, not only the failing slice.
- Model interventions
- Router threshold changes, model swaps, or preference tuning (DPO) on failure pairs where the correct behaviour is known and stable.
- Pipeline interventions
- Chunking and hybrid retrieval with a reranker for grounding defects; constrained decoding for schema defects; typed tool contracts for tool-call errors.
- A/B and interleaved tests
- A/B tests for CTQ proportions with sample size from the baseline p̄. Team-draft interleaving for retrieval and ranking changes, which reaches a preference verdict on far less traffic.
Control so the gain holds after the project team leaves.
A control plan names, for each CTQ, the chart, the rules evaluated, the owner and the reaction. Limits are recomputed only after a verified process change, with a fresh Phase I baseline — never silently re-fitted to absorb a drift.
- Control charts
- p-chart for the daily defect proportion with per-day limits for varying sample size; EWMA for latency and cost. Western Electric rules 1, 2 and 4 on every point.
- Automated alerting
- Rules 2 and 4 open a ticket with the chart, the rule and the change log for the window. Rule 1 pages on-call with a runbook link.
- Guardrail regression gates
- In CI, a deploy is blocked when any CTQ on the golden set is worse than production at α = 0.05. At runtime, a Rule 1 signal pins the router to the last-known-good configuration hash.
- Heightened inspection
- While a gate is engaged, human-review sampling rises from 5% to 25% of decisions until the chart shows eight consecutive in-control days.
Control
A control chart that pages someone before customers notice.
A simulated production model scored on 800–1,400 sampled decisions a day. The process starts in control; from day 41 its true defect rate creeps upward by 0.05 points a day. Watch which rule catches it first.
- Daily defect proportion pᵢ = defectsᵢ / nᵢ
- Centre line p̄, pooled over days 1–30
- 3σ limits, stepped per day as nᵢ varies (800–1,400)
- R1 — action: point beyond 3σ
- R2 / R4 — early warning: 2 of 3 beyond 2σ, or 8-point run
Alert log
- No alerts. Rules 1, 2 and 4 evaluated on every appended day.
Data table (40 days)
| Day | n | Defects | p | UCL | LCL | Rules |
|---|---|---|---|---|---|---|
| 1 | 1066 | 12 | 0.0113 | 0.0183 | 0.0005 | — |
| 2 | 1247 | 16 | 0.0128 | 0.0176 | 0.0012 | — |
| 3 | 922 | 12 | 0.0130 | 0.0190 | 0.0000 | — |
| 4 | 1188 | 12 | 0.0101 | 0.0178 | 0.0010 | — |
| 5 | 936 | 5 | 0.0053 | 0.0189 | 0.0000 | — |
| 6 | 850 | 6 | 0.0071 | 0.0193 | 0.0000 | — |
| 7 | 996 | 6 | 0.0060 | 0.0186 | 0.0002 | — |
| 8 | 989 | 9 | 0.0091 | 0.0186 | 0.0002 | — |
| 9 | 1161 | 14 | 0.0121 | 0.0179 | 0.0009 | — |
| 10 | 1119 | 12 | 0.0107 | 0.0181 | 0.0008 | — |
| 11 | 855 | 8 | 0.0094 | 0.0193 | 0.0000 | — |
| 12 | 1012 | 9 | 0.0089 | 0.0185 | 0.0003 | — |
| 13 | 1192 | 13 | 0.0109 | 0.0178 | 0.0010 | — |
| 14 | 1300 | 11 | 0.0085 | 0.0174 | 0.0014 | — |
| 15 | 935 | 5 | 0.0053 | 0.0189 | 0.0000 | — |
| 16 | 1246 | 13 | 0.0104 | 0.0176 | 0.0012 | — |
| 17 | 1158 | 9 | 0.0078 | 0.0179 | 0.0009 | — |
| 18 | 1166 | 12 | 0.0103 | 0.0179 | 0.0009 | — |
| 19 | 1143 | 7 | 0.0061 | 0.0180 | 0.0008 | — |
| 20 | 973 | 8 | 0.0082 | 0.0187 | 0.0001 | — |
| 21 | 1084 | 10 | 0.0092 | 0.0182 | 0.0006 | — |
| 22 | 1314 | 13 | 0.0099 | 0.0174 | 0.0014 | — |
| 23 | 992 | 3 | 0.0030 | 0.0186 | 0.0002 | — |
| 24 | 1115 | 13 | 0.0117 | 0.0181 | 0.0007 | — |
| 25 | 841 | 7 | 0.0083 | 0.0194 | 0.0000 | — |
| 26 | 1049 | 11 | 0.0105 | 0.0184 | 0.0005 | — |
| 27 | 1195 | 8 | 0.0067 | 0.0178 | 0.0010 | — |
| 28 | 1009 | 11 | 0.0109 | 0.0185 | 0.0003 | — |
| 29 | 1357 | 16 | 0.0118 | 0.0173 | 0.0015 | — |
| 30 | 1317 | 17 | 0.0129 | 0.0174 | 0.0014 | — |
| 31 | 1350 | 19 | 0.0141 | 0.0173 | 0.0015 | — |
| 32 | 1184 | 13 | 0.0110 | 0.0178 | 0.0010 | — |
| 33 | 854 | 7 | 0.0082 | 0.0193 | 0.0000 | — |
| 34 | 1161 | 10 | 0.0086 | 0.0179 | 0.0009 | — |
| 35 | 816 | 11 | 0.0135 | 0.0196 | 0.0000 | — |
| 36 | 1249 | 13 | 0.0104 | 0.0176 | 0.0012 | — |
| 37 | 1077 | 5 | 0.0046 | 0.0182 | 0.0006 | — |
| 38 | 831 | 7 | 0.0084 | 0.0195 | 0.0000 | — |
| 39 | 1309 | 12 | 0.0092 | 0.0174 | 0.0014 | — |
| 40 | 1337 | 14 | 0.0105 | 0.0173 | 0.0015 | — |
Why a p-chart
Each decision is defective or not, and the number of decisions scored varies by day. The p-chart's limits, p̄ ± 3·√(p̄(1−p̄)/nᵢ), widen on light days and tighten on heavy ones, so a noisy Sunday is not mistaken for a shift.
Why the limits are frozen
p̄ comes from the day 1–30 Phase I baseline and then stays fixed. Re-fitting the centre line on every new point lets a slow drift carry the limits upward with it, and the chart never alarms.
Why three rules
Rule 1 (one point beyond 3σ) is the action signal. Rule 4 (eight consecutive points on one side) and Rule 2 (two of three beyond 2σ) fire earlier on small sustained shifts, which is how drift actually arrives.
Instrumentation
The monitoring stack we install.
Six instruments, each producing a signal with an owner and a reaction. Together they close the loop from a code change to a production alert to a verified fix.
- Decision
Eval harness in CI
Every prompt, model, retrieval or tool change runs the CTQ suite as a required check. The build fails when any CTQ regresses beyond tolerance against current production.
Signal · Per-CTQ pass rate, diffed against prod
- Inference
Shadow traffic
Candidate configurations receive mirrored production requests. Outputs are scored and ledgered, never returned to users, so we see real-distribution behaviour before exposure.
Signal · Defect rate delta, candidate vs prod
- Data
Golden sets
Versioned, stratified, human-labelled cases per CTQ, with a sealed holdout that is never used for prompt iteration so the suite cannot be overfit.
Signal · Judge–human agreement (κ) per release
- Data
Drift monitors
Population Stability Index on intent mix and embedding-cluster assignment; KL divergence on output labels against a rolling 28-day reference window.
Signal · PSI > 0.2 or KL above its own control limit
- Guardrail
Cost & latency SPC
p-charts for defect proportion; EWMA charts for p95 latency and cost per decision, which catch small sustained shifts that Shewhart limits miss.
Signal · Western Electric rules 1, 2, 4
- Guardrail
Alert routing to on-call
Warnings open a ticket with the chart, the rule, and the change log for the window. Rule 1 pages on-call and flips the router to the last-known-good configuration hash.
Signal · Time-to-acknowledge, time-to-contain
Put your AI systems under statistical control
The Systems Assessment inventories every model and decision path you run, drafts CTQ defect definitions with your owners, and specifies the charts, rules and gates for each — ready to instrument.