Handover
Runbooks
Symptom-indexed procedures for every alert — guardrail flag spike, drift breach, provider outage, chain-verification failure — each rehearsed once in a game day your team leads.
Engineering
Forward Deployment, Context, Goal & Loop Engineering
Four disciplines that decide whether an AI system survives contact with production: who builds it and where, what the model is allowed to see, what it is actually optimising, and how it gets better without drifting.
01 · Forward Deployment Engineering
Engineers at the edge of your org, shipping into your environment
No slide decks handed over at the end. Our engineers join your repositories, your change process and your on-call rotation, and build inside your cloud accounts under the controls you already audit.
Access model
- Identity
- Named accounts in your IdP with MFA enforced. Elevated roles are granted just-in-time through your PAM tooling against a ticket reference and expire within the working day.
- Devices
- We work from your VDI or managed laptops enrolled in your MDM. No client data is stored on Earp-owned devices.
- Change
- Every change is a pull request through your CI/CD and your code-owner review. We hold no deploy rights your own engineers do not.
- Data
- Read access through masked views or governed shares scoped to the use case. Production writes happen only through the application's own service identity.
- Offboarding
- Access ends when group membership is removed in your IdP. The handover pack includes an attestation listing every grant we held and when it was revoked.
Milestones
- Week 1
Access, baseline, first ledger entry
- Accounts provisioned through your IdP; decision log opened in your repository.
- One production route instrumented end to end — traces plus a ledger write — before any model logic changes.
- Baseline captured: latency, cost per decision, override rate, current failure taxonomy.
- Week 4
First governed release in canary
- Golden eval set agreed with the domain owners, with a written labelling guide.
- Guardrail bundle for the route running in CI with the red-team suite.
- First release hash promoted to a small canary ring behind an eval gate; rollback rehearsed.
- Week 12
Your team runs it without us
- Governed route on full traffic with control charts, owners and paging.
- Runbooks exercised in a game day led by your engineers, not ours.
- Handover signed off against the artefact checklist below.
Handover artefacts
What stays when we leave. Each artefact lives in your repositories and is exercised by your team before sign-off.
Handover
Eval suites
Golden, adversarial and regression sets versioned with the release and runnable in your CI with one command, shipped with the rubric and labelling guide that produced them.
Handover
Infrastructure-as-code
Terraform modules for networking, private endpoints, keys and log sinks, with policy checks at plan time and drift surfaced in the same report your platform team already reads.
Handover
Architecture decision records
One ADR per consequential choice — model selection, retrieval topology, escalation thresholds — recording the alternatives rejected and the evidence that decided it.
02 · Context Engineering
The context window is a budget, not a bucket
What the model sees bounds what it can do. We allocate the window like a budget: every segment has a ceiling, every token has provenance, and overflow is resolved by relevance — never by truncating whatever arrived last.
Context window
Synthetic demo data0 / 96k0%
- System prompt
- Policy
- Retrieved docs
- Conversation
- Tool results
- Compressed (hatched)
- Evicted (fades out)
Window empty. Press Play or Step to admit 30 segments into a 96k budget.
- Resident segments
- 0
- Prune events
- 0
- Tokens reclaimed
- 0
Eviction log (latest first)
Nothing pruned yet. Segments with relevance ≥ 0.60 are compressed to a quarter of their size before anything above them is evicted; system prompt and policy are pinned.
The demo replays a fixed, seeded stream of 30 segments. On overflow it visits resident segments in ascending relevance: segments at or above the floor are compressed once to a quarter of their size, the rest are evicted. Conversation turns lose relevance with age. Change the budget and the same stream is replayed under the new ceiling.
Window budgeting
Per-segment token ceilings — system, policy, evidence, history, tool output — are declared per route; the packer fills in priority order and resolves overflow by relevance, never by cutting the tail.
Retrieval topology
Flat, hierarchical (section → chunk) or graph-expanded retrieval is chosen per corpus by measured recall@k on a labelled query set, not by whichever default the framework ships.
Provenance chains
Each chunk carries source URI, document version hash and byte offsets; the ordered list is hashed into a manifest on the ledger entry, so an answer traces to the exact bytes that informed it.
Context compression
Segments above the relevance floor are compressed extractively — query-keyed sentence selection — before anything is evicted, and each summary keeps a pointer back to its full source.
Find out what your models are actually seeing
The Systems Assessment includes a context audit: we capture real prompts from each production route, attribute every token to a source, and show you where budget is spent on content that never influences the answer.
03 · Goal Engineering
Business outcomes as machine-checkable objectives
“Improve claims handling” is not something a model or an eval can check. We decompose each outcome into a metric tree, bind every metric to predicates a program can evaluate, gate releases on them — and then assume the system will try to game them.
Reward hacking, and what we do about it
Any objective strong enough to steer a system is strong enough to be exploited by it. Each failure mode gets a named countermeasure and a signal that tells you it is happening.
| Failure mode | How it shows up | Countermeasure | Detection signal |
|---|---|---|---|
| Metric gaming | The system optimises the measured proxy — short handle time — by deferring or escalating the hard cases. | Held-out adversarial evals nobody tunes against, and paired metrics that move in opposite directions under gaming (cycle time with reopen rate). | Paired metric diverges; adversarial pass rate falls while golden-set score rises. |
| Specification gaming | Output satisfies the letter of the objective and violates its intent — a denial that cites a clause that does not apply. | Constraint predicates evaluated beside the objective (the cited clause must entail the stated reason); a violation fails regardless of score. | Constraint violations on shadow traffic. |
| Goodhart drift | Scores climb on a static golden set as prompts and retrieval are gradually tuned to it. | Rotating golden sets: a fixed share of cases is retired each cycle and replaced from fresh production samples, with the rotation ledgered. | Gap between retired-set and fresh-set scores widens. |
| Eval contamination | Eval cases leak into few-shot examples, fine-tuning data or the retrieval index. | Content hashes of every eval case are intersected with prompt, training and index manifests in CI; any overlap fails the build. | Non-empty hash intersection. |
| Judge exploitation | An LLM judge rewards length, confident tone or the house style of its own model family. | Judges calibrated against human labels per rubric item, length-controlled scoring, and a judge from a different model family than the system under test. | Judge–human agreement (Cohen's κ) drops below the calibration floor. |
| Reviewer sycophancy | The system learns phrasing that reviewers approve rather than answers that are correct. | Seeded known-wrong cases in every review queue and blind samples with outputs rewritten in neutral register. | Any approval of a seeded known-wrong case. |
04 · Loop Engineering
Observe, hypothesize, act, evaluate, ledger — then again
Production AI systems do not improve on their own; left alone they drift. We install a closed loop with a fixed cadence and pre-registered hypotheses, so every change is a measured experiment and every result — including the failures — lands on the ledger.
Pull traces, ledger entries, reviewer overrides and drift statistics for one route over a fixed window, sliced by segment, so the loop starts from measured behaviour rather than the loudest complaint.
ledger ← observation_window_id
Pre-register the change, the metric it should move, the threshold and the sample size before anything is built — e.g. deeper re-ranking cuts unsupported claims by 30% without pushing p95 latency past 2.5 s.
ledger ← hypothesis_id, predicted_effect
The change ships as a new release hash into a shadow or canary ring with every other component pinned, so any movement in the metrics is attributable to exactly one diff.
ledger ← release_hash, deployment_ring
Golden set, adversarial set and live canary traffic are compared with the incumbent using a sequential test that can only stop early at boundaries declared in the hypothesis.
ledger ← eval_run_id, verdict
Accepted and rejected hypotheses are both appended with their evidence. The next observation window starts from the new baseline, and nobody re-tests an idea the ledger already refuted.
ledger ← loop_iteration, baseline_hash
Fixed cadence
Each governed route runs one loop iteration per agreed cycle, so improvement is scheduled work with an owner, not a reaction to the latest incident.
One diff per iteration
Act ships exactly one change as a new release hash with everything else pinned, which is what makes the evaluation attributable.
Stop rules declared up front
Sample size, success threshold and early-stopping boundaries are written into the hypothesis before the change is built, so results cannot be peeked into significance.
Book a Systems Assessment
Two weeks, fixed scope. We map every model, prompt, data flow and decision path you run today, score them against your compliance regime, and hand you a ranked remediation plan with named mechanisms — not a slide deck.