​Your AI Demo Works. Your AI Product Doesn't. The Harness Is Why

Vatsal Bhardwaj - Founder and CEO, Jabali AI. Expert in AI, Gaming and Cloud Computing.

gettyEvery executive has lived some version of this scene. A team gathers around a screen. Someone types a prompt. The model does something remarkable—drafts the contract, triages the ticket, writes working code—and the room decides the future has arrived. A budget appears. A pilot launches.

The numbers say this is normal. MIT researchers found that 95% of enterprise generative AI pilots deliver no measurable P&L impact, and Gartner expects more than 40% of agentic AI projects to be canceled by the end of 2027. The comfortable explanation is that the models aren’t ready yet. After two years of shipping autonomous AI systems in one of the most unforgiving domains I know, I’ve come to believe the comfortable explanation is also the expensive one.

The space between a dazzling demo and a dependable product is where most enterprise AI investments now go to die, and bigger models don’t close it.

The harness is everything you build around the model: the context it sees, the tools it can use and the sandboxes it acts in. It is also the workflows that sequence its work, the memory that grounds it in your business and the evaluation loops that tell you whether the output is any good before your customer finds out.

Demos don’t need harnesses. A demo is one attempt, on a forgiving example, judged by an audience rooting for the magic. A product is ten thousand attempts a day, on real customers’ messy intent, judged by people who don’t care how impressive the technology is, only whether the thing works. Demos are graded on the best case. Products are graded on the worst one.

My company builds AI systems that turn a plain-language idea into a playable video game, and I’ve spent 15-plus years building games and platforms for very demanding customers. Games are an honest proving ground for AI because they are objective and subjective at once. A confident paragraph of text can be wrong in ways a reader never notices; an unplayable game fools nobody. The code has to compile, the build has to boot and the art, story and mechanics have to agree with one another. And then the whole thing has to survive the hardest question in software: Is it fun?

That severity has forced us to rebuild our harness five times in 24 months: a rigid orchestrator first, then a human-approval copilot, then structured generation pipelines, then a graph of specialized agents and now fully autonomous agents writing real code in isolated sandboxes. Each rebuild taught a lesson that applies well beyond gaming:

• Structure beats freestyle. When you control the scaffold, models stop inventing architectures and spend their capability on craft instead.

• How agents hand off work to one another shapes quality as much as raw model ability does. We now treat orchestration as a product surface and design it deliberately.

• Autonomous agents need walls. Sandboxes are what let ours take big swings in real code without taking down anything that matters.

• Never marry a model. The best model for our workload has changed repeatedly, and each switch has gotten cheaper because the harness absorbed the change.

The deeper finding in the MIT research was that pilots stall because systems don’t learn from feedback and don’t fit real workflows.

The old rule that you can’t improve what you don’t measure has a harsh corollary in AI: for most valuable business tasks, the measurement doesn’t exist yet and you have to invent it. We grade every output in layers. Objective checks come first (does it run?), then coherence checks (do the pieces agree with each other and with the request?) and only then subjective quality, scored by AI judges we calibrate against human ratings.

Early on, our characters would not stay in character. A gruff blacksmith written to grunt a few words about ore would, ten minutes into a playtest, brighten into a cheerful assistant offering the player a numbered list of quest options. Nothing errored, and every metric we had was green. We caught it because someone on the team spent a week reading raw transcripts and noticed that every character in the game was slowly converging on the same friendly voice. We named it persona drift, wrote checks for it and we have watched that number on every release since. That is the general pattern: humans discover new failure modes by reading transcripts, and once a failure has a name, it becomes a metric. No model gets promoted into production because it is exciting; it runs the suite like everything else, and the results make the call.

1, Rebalance the budget toward the machinery. Model API costs should be a minority of your AI spending. The majority belongs to context, tooling, integration and evaluation, because those are what make intelligence dependable enough to charge for.

2. Before you scale, write down what “correct” means for your use case, then what “good” means, in that order. If you can’t grade it, you can’t ship it.

3. Put your best people on error analysis before you build a single dashboard. Have them read raw outputs weekly; that’s where failure modes get discovered and named, and the metrics follow from the names.

4. Stay model-agnostic. Swapping models should cost a config change and an eval run rather than a re-platforming project, so when a new model ships, your suite can tell you within days whether it earns a place in production.

Everyone can get their hands on a frontier model now. The harder part is building the systems around it that make AI useful day after day. That’s where a demo starts becoming a real business capability. Your customers will never see your model. They will only ever experience your harness.

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?