Getting an AI system from a promising demo into production, where it touches revenue and real customers and can no longer fail quietly, is a different discipline. I've practiced it twice: for 300 million people at Indeed, and alone, as a founder with paying customers. I do that now for companies that are done experimenting.
Somewhere in your company is a pilot that impressed everyone in March and hasn't shipped since. It performs beautifully in a sandbox. No customer has ever touched it. Nobody will kill it, because it works, more or less. Nobody will ship it, because no one can say what happens when it is wrong at volume.
The pattern behind MIT's number is remarkably consistent, and it has almost nothing to do with which model anyone chose. Four habits, over and over:
The assistant that can't act. It chats. It cannot read the CRM, write to the ticketing system, or touch the order flow. Impressive in a meeting; useless on a Tuesday.
Launches on faith. No evaluation harness, no defined bar a change must clear, no way to know whether last month's update made things better or quietly worse.
One bot for everything. The general-purpose company assistant is almost always the wrong shape. What survives contact with production: narrow agents, specific workflows, a human somewhere in the loop.
Ownership by committee. AI lives with an innovation team, three vendors, and a steering group. Pilots multiply. Nothing ships.
What breaks the stall is not another pilot. It is judgment: someone who has shipped this before, looking at what you have and putting one of three words on it, in writing.
Wrong abstraction, no data path, no owner. Ending them early is the cheapest decision in all of AI. I put it in writing and say it to your face, in week two rather than quarter three.
The common verdict. A stalled pilot usually sits three or four specific blockers from shipping: evaluation, integration, governance. The report names yours and puts them in order.
Occasionally the foundations are sound and the hesitation is instinct rather than evidence. Then the job is speed: launch gates, guardrails, a kill switch, so the system stays trustworthy at ten times the traffic.
Six questions from the first ten minutes of a real audit. Answer honestly; nobody is watching, and the verdict updates as you go.
Is any AI system you've built handling real work in production today — not a sandbox, not a beta?
Can your AI read from and write to your systems of record (CRM, ERP, ticketing) — or does it only chat?
Could anyone on your team prove, with data, whether your AI got better or worse last month?
Is there a defined bar an AI change must clear before it ships — and a kill switch if it misbehaves after?
Does a single senior executive own AI outcomes — not a committee, not "innovation"?
If your AI vendor tripled prices tomorrow, could you swap models without rebuilding?
Awaiting answers…
Six questions, no email required. The stamp appears when you finish.
Fixed scope and fixed fee, agreed before work starts. I deliver every engagement myself.
Two to three weeks inside your pilots: architecture, data paths, evaluation coverage. Each pilot receives a written verdict with the reasoning shown, plus a readiness scorecard and a 90-day sequence for the survivors.
The flagship. An evaluation harness, defined launch gates, behavioral guardrails, and a kill switch, installed and running, with your team trained to operate them. Modeled on the machinery I built at Indeed to decide what ships to 300 million people.
For product companies putting agents into the product itself. Agent design, human-in-the-loop mechanics, build-or-buy decisions, and an architecture record your engineers can execute without me in the room.
A senior operator inside your leadership team two or three days a week: roadmap, hiring bar, org design, architecture calls, board reporting. Two seats exist. The six-month minimum is deliberate; nothing real happens faster.
AI oversight for boards, and per-deal diligence for investors who need to know whether a target's AI claims survive a look at the actual architecture. They often don't, and it is better to learn that before wiring the money.
Twenty-five years, condensed to what shipped. Context on any of these, gladly, on a call.
people served by the AI matching and launch-governance systems built at Indeed. The governance machinery still decides what ships there.
reduction in bad matches after LLM + RLHF matching went live. Recommendation conversion rose by half.
in annual revenue carried through a hard regulatory deadline at Amazon Canada, on a catalog-wide ML compliance program.
agentic AI shipped to production. Once with a 300-person org behind it, once entirely alone. Both have paying customers.
Software engineer to engineering manager. Nine years building systems where being wrong costs money.
Took Proofhub, a SaaS product, from nothing to $2M in annual revenue. Product, engineering, and sales in one seat.
Fraud detection at PayPal's transaction volume. Tripled engagement on Bing's knowledge graph. Ran payments for one of Asia's largest travel marketplaces: 90+ local payment methods, cross-border rails in five countries.
Founded the technology org and grew it past a hundred people. ML compliance across millions of listings. Launched BNPL at marketplace scale.
LLM + RLHF matching for 300 million people, and the governance system that still gates every AI release. Also switched off a legacy channel that was quietly damaging the marketplace while adding very little to it, saving $24M a year.
A live agentic fintech product with paying customers, designed, written, and operated end to end with AI agents, by one person. Depilot is the practice built from what both scales taught.
Three scenarios in detail. In all three the broken thing was a measurement, not a model — and in all three the fix ran through multi-model councils and human-in-the-loop designed on purpose rather than assumed.
01A written bar every AI change had to clear, and no way to tell whether a change cleared it. Building a golden set from real traffic, replacing the single LLM judge with a three-model council, and routing every disagreement to a human whose verdict is written back.
A support copilot reporting 42% deflection was counting abandonment as success, and the metric improved whenever the bot got harder to escape. The honest number was 19% — and a second model now checks every irreversible action before it executes.
Fourteen months in, operations still reviewed every prediction by hand — because the confidence score carried no information. Calibration, cost-weighted thresholds, and a second opinion on the 0.6% of documents holding most of the risk. No retraining.
Illustrative scenarios — composites of patterns seen repeatedly, not named engagements. The companies are invented; the failure modes and fixes are real. All three case studies →
Five gates stand between a demo and a system you would bet revenue on. Every stalled pilot is stopped at one of them — and almost never the one the team expects. Each gate has an artifact that either exists or doesn't, which is what makes this an assessment rather than an opinion.
The question the 95% never answered. A pilot with no number attached has no basis for the production spend, and dies quietly at budget time.
ArtifactA baseline metric it is supposed to move, and the current value of that metric.
Gate openNobody can state the number without a meeting first.
Reading is half a system. Until it can write back — create, update, trigger — every outcome still depends on a human retyping it.
ArtifactRead and write access to the systems of record the work lives in.
Gate openIt chats. Impressive in a meeting, useless on a Tuesday.
The test oracle: the thing that says whether an output is correct. Without one there is no evaluation, no regression detection, and no way to compare two versions. Every other gap downstream is really this one.
ArtifactA labelled golden set from real traffic, and graders that run against it unattended.
Gate open"We test it manually." Someone tries five prompts and forms an impression.
Not for uptime — for whether the thing is any good. This is usually why measurement never gets funded: it is nobody's number to miss.
ArtifactOne named senior executive, with the outcome in their objectives.
Gate openA committee, a steering group, or "innovation".
A floor is only real if something enforces it. A written bar with no measurement behind it approves everything that reaches it, and records that it did.
ArtifactThresholds wired into the release path that fail a build, plus a kill switch a non-engineer can pull.
Gate openThe gate has never once blocked a release.
How the method is applied
Bring the stuck pilot. I work the gates in order and find the one you're actually stopped at. Plenty of callers discover they only need to close one, and go and do it themselves.
Two to three weeks inside the systems, testing each gate against what is actually there rather than what the team believes is there. Every pilot leaves stamped kill, fix or scale.
Oracle and Floor are the two gates you cannot close with a document. Golden set, graders, thresholds in the release path, guardrails, kill switch — running in your stack and operated by your team.
The method is the same object at every depth: the pre-audit above scores it, the audit evidences it, the install builds it. Score yourself in 60 seconds →
You work with me. There is no bench and no handoff after the sale. That limits how many clients I can take, which is the point.
Engagements end with something running. A harness, a gate, a system your team operates. Documents describe the work; they aren't the work.
Prices are fixed and quoted up front. I don't bill you for thinking slowly.
Bad news arrives early. If the right answer is to kill the project, you hear it in the first two weeks, while it's still cheap.
I run the practice on AI agents. Research, drafting, analysis, with the judgment kept human. Ask to see how it works; it doubles as a demo.
I wrote my first production code at Bloomberg in 2000, and spent the next twenty-five years as the person difficult systems got handed to. I founded Amazon Canada's technology organization, grew it past a hundred people, and carried five billion dollars of revenue through a regulatory deadline that had no extension in it.
At Indeed I ran search and recommendations for 300 million people. My teams shipped the LLM matching that cut bad matches by ninety percent, and built the evaluation and launch-governance machinery that still decides what ships there. I also switched off a legacy channel that was quietly damaging the marketplace while adding very little to it, which saved twenty-four million dollars a year. Nobody had looked hard at it in years. That decision is this practice in miniature.
In 2025 I left to test a private conviction: that one operator with disciplined AI agents could build what used to take a team. One More Million, a live fintech product with paying customers, is the result. Depilot is the practice built on what both scales taught me.
"Most AI advice fails in the gap between a demo that impresses and a system you'd bet revenue on. I've crossed that gap twice. The crossing is the practice."
Thirty minutes with someone who has shipped what you're attempting. You'll leave knowing why it stalled and what shipping would take. That holds whether or not I ever send you an invoice.