Control Charts for Language Models

Most AI quality dashboards answer the wrong question. They show an accuracy line that wobbles and a red threshold someone picked in a planning meeting. The wobble is mostly noise, the threshold was never derived from that noise, and so the alert fires on Saturdays for no reason and stays silent through a two-week slide that quietly doubles the defect rate.

Manufacturing solved this problem in the 1920s. Walter Shewhart's insight was that every process has inherent variation, and monitoring exists to separate that routine variation from variation with an assignable cause. A language model in production is a process: inputs arrive, outputs are inspected, some fraction fail. The machinery that governs a stamping press applies unchanged, and it is far better at telling you when something actually changed than any threshold.

Define a defect before you chart one#

Control charts need a count of discrete failures per inspected unit. "Quality" is not a count. You need operational definitions applied the same way every day. Four sources cover most production systems:

• Eval rubric failures. A fixed daily sample scored against a versioned rubric by a calibrated LLM judge, humans, or both. An item fails if it misses any critical criterion.
• Guardrail blocks. Every request or response the enforcement layer rejected: injection, PII egress, policy denial on a tool call.
• Schema violations. Structured outputs that fail the downstream contract, counted before repair or retry logic masks them.
• Escalations. Items routed to human review on low confidence, or where a reviewer overturned the model.

Two rules keep these honest. Chart each defect class separately, because mixing classes averages away the signal. And freeze the definition: a rubric change is a deliberate process change, recorded and re-baselined like any other. A chart whose measurement drifts is measuring the measurement.

The daily aggregation is plain SQL over the decision event table. Note the two denominators: guardrail and schema checks run on all traffic, rubric evaluation runs on a sample.

SELECT
  date_trunc('day', created_at AT TIME ZONE 'UTC')        AS day,
  route,
  COUNT(*)                                                AS n_all,
  COUNT(*) FILTER (WHERE NOT schema_valid)                AS schema_violations,
  COUNT(*) FILTER (WHERE guardrail_verdict = 'block')     AS guardrail_blocks,
  COUNT(*) FILTER (WHERE escalated)                       AS escalations,
  COUNT(*) FILTER (WHERE in_eval_sample)                  AS n_sampled,
  COUNT(*) FILTER (WHERE in_eval_sample
                     AND NOT rubric_pass)                 AS rubric_failures,
  array_agg(DISTINCT model_version)                       AS model_versions,
  array_agg(DISTINCT retrieval_index_id)                  AS index_builds
FROM ai_decision_events
WHERE created_at >= now() - interval '120 days'
GROUP BY 1, 2
ORDER BY 1, 2;

The last two columns pay for themselves the first time a signal fires: each subgroup carries the component versions that produced it.

The p-chart, with honest denominators#

A p-chart plots the fraction defective per subgroup. Here a subgroup is one day of one route. Because traffic and sample sizes vary by day, the limits vary by point. With p_i = d_i / n_i for day i, the center line is the pooled baseline rate p̄ = Σd / Σn, and the limits are p̄ ± 3·sqrt(p̄(1−p̄)/n_i), with the lower limit floored at zero. Small-sample days get wide limits; heavy days get tight ones. This is the first thing a fixed threshold gets wrong: it applies the same bar to a 150-item Sunday and a 400-item Tuesday.

Two refinements matter for model traffic.

Standardize for run rules. Because limits move, run rules are easiest to apply to the standardized value z_i = (p_i − p̄) / sqrt(p̄(1−p̄)/n_i). Every point then lives on the same scale, where the control limits sit at ±3 and the zone boundaries at ±1 and ±2.

Watch for overdispersion. On all-traffic counts, n_i runs into the tens of thousands and binomial limits become razor thin. Real day-to-day variation in input mix exceeds binomial noise, so nearly every point signals. If many baseline points fall outside limits with no assignable cause, switch to a Laney p′ chart, which scales the binomial sigma by the average moving range of the z-scores divided by 1.128, absorbing between-day variation you cannot remove.

Catching drift before the limits break#

A point beyond three sigma is the loudest signal and the least common. Most production regressions are shifts of one to two sigma that never cross a limit. The Western Electric rules, and Lloyd Nelson's later extended set, detect these from patterns in consecutive points, each with a low false-alarm rate on a stable process.

import numpy as np

def p_chart(d, n, baseline=slice(0, 25)):
    d, n = np.asarray(d, float), np.asarray(n, float)
    p_bar = d[baseline].sum() / n[baseline].sum()
    sigma = np.sqrt(p_bar * (1 - p_bar) / n)
    ucl, lcl = p_bar + 3 * sigma, np.clip(p_bar - 3 * sigma, 0, None)
    return p_bar, ucl, lcl, (d / n - p_bar) / sigma

def run_rules(z):
    hits = []
    for i in range(len(z)):
        def win(k):
            return z[i - k + 1 : i + 1] if i >= k - 1 else None
        if abs(z[i]) > 3:
            hits.append((i, "WE1 beyond 3 sigma"))
        for side in (1, -1):
            w3, w5, w9 = win(3), win(5), win(9)
            if w3 is not None and (side * w3 > 2).sum() >= 2:
                hits.append((i, "WE2 two of three beyond 2 sigma"))
            if w5 is not None and (side * w5 > 1).sum() >= 4:
                hits.append((i, "WE3 four of five beyond 1 sigma"))
            if w9 is not None and (side * w9 > 0).all():
                hits.append((i, "Nelson 2 nine on one side"))
        w6 = win(6)
        if w6 is not None:
            step = np.diff(w6)
            if (step > 0).all() or (step < 0).all():
                hits.append((i, "Nelson 3 six-point trend"))
    return hits

Each rule points toward a different class of cause, which makes the chart diagnostic rather than just an alarm.

An unplanned improvement is a special cause too, most often instrumentation failure.

Common cause, special cause, and the ledger#

Common-cause variation is the noise of a stable system: the model's inherent error rate against the current input population. Reacting to individual common-cause points (tuning a prompt because Tuesday was bad) adds variation rather than removing it; Deming called this tampering. Special-cause variation has an assignable source: a model version bump, a retrieval index rebuild, a template change in the upstream documents the system reads.

The chart tells you that something changed and roughly when. It cannot tell you what. That requires a timestamped record of every change to every component, which is where process control meets a decision ledger. Every deliberate change (model snapshot, prompt hash, index build, rubric version, policy bundle) is a ledger entry. Every special-cause signal is also a ledger entry: chart, rule, subgroup date, the change events within the look-back window, the disposition (cause found, cause not found, measurement error), and the action taken. Over time that pairing becomes the evidence base for your change-management process, and it maps directly onto the MEASURE and MANAGE functions of the NIST AI RMF. We cover the record structure in the ledger and the broader quality practice in quality engineering.

Control limits describe the process you have. Specification limits describe the process you want. Never draw one where the other belongs.

A requirement that rubric failures stay under 3% is a specification. It belongs on the chart as a separate line answering "is the process capable?" Control limits answer "is the process stable?" Conflating the two is why threshold alerting is simultaneously too noisy and too slow.

Phase I, phase II, and re-baselining#

Charts run in two phases. In phase I, you collect twenty to thirty subgroups from a period you believe is stable, compute trial limits, investigate every point that signals, remove those with confirmed assignable causes, and recompute. You repeat until the baseline is clean. The output is a frozen p̄ and a documented baseline window. In phase II, those limits are fixed and new points are judged against them. Phase II limits do not update with each new day. A rolling baseline would slowly absorb the very drift you are trying to detect.

Re-baselining is legitimate only after a deliberate, recorded change: a new model snapshot, rubric version or retrieval design. Log the change, expect a shift on the old chart, then run a fresh phase I on post-change data before freezing new limits. Evaluating canary traffic in parallel shortens this. What you must never do is re-baseline to make an alarm stop. If the chart is signaling and you cannot name the change that caused it, you have an investigation, not a new normal.

A worked example#

The following is hypothetical. Suppose a claims-intake pipeline extracts fields from about 12,000 documents a day and sends a random 400 of them (150 on weekends) to rubric evaluation. Phase I across 25 clean days gives p̄ = 0.021.

On a weekday, sigma = sqrt(0.021 × 0.979 / 400) ≈ 0.0072, so the upper limit is about 4.25%, or 17 failures. On a weekend it widens to about 5.6%. Compare the fixed threshold the team had been using, alert above 3%: on a 150-item sample, a stable 2.1% process produces five or more failures roughly one weekend day in five. The threshold pages people for noise.

Then a payer changes its claim form template on a Tuesday. One field starts extracting incorrectly on a subset of documents, and the true defect rate moves to about 3.0%. On weekday samples that is z ≈ 1.25: no single day approaches the 4.25% limit. But a sustained run of points above one sigma is exactly the shape WE3 and Nelson 2 detect, and for a shift this size one of them typically fires within the first week and very likely within two. The ledger shows no model or index change in that window, which redirects the investigation to inputs, and stratifying by payer finds the template. Without the run rule, a shift of 0.9 points across 12,000 daily documents is roughly 108 additional defective extractions a day, accumulating for as long as the dashboard looks "fine."

What to do on Monday#

• Write operational definitions for each defect class, version them, and store the version with every evaluated item.
• Fix the daily eval sample size per route and draw it randomly; convenience samples break the binomial model.
• Build the daily aggregation above, including component versions per subgroup.
• Run a phase I on the last 25 to 30 clean days per route and defect class. Freeze the limits and document the baseline window.
• Implement WE1 through WE3 plus Nelson 2 and 3 on standardized values. Route signals to the owning team, not a shared channel.
• Log every signal and every deliberate change to the same ledger, and require a disposition on each signal within a set number of days.
• Draw your SLA as a separate specification line, and stop paging on it alone.