How to tell a real AI demo from a scripted one

Every AI demo works. That's the problem — it tells you nothing, because a demo is a path someone walked a hundred times before you saw it.

What separates a system that's running from one that's rehearsed isn't visible in the happy path. It's visible in the parts nobody demos: the query nobody planned for, the logs, the invoice.

Ask for a query they didn't plan

The simplest test. Take the demo off its rails: hand them a phrase from your own catalog, in your own customers' words. Not "show me red shoes" — something oblique. "Something for standing all day." "Will this survive a toddler."

What you're watching for isn't whether it's right. It's whether it's confident and wrong, which is the failure mode that costs you money. A system that says "I'm not sure, here are three things that might fit" is better engineered than one that picks something and commits.

Our own search ranked sweatpants first for "breathable for a hot summer day" the first time we asked it something we hadn't rehearsed. The scores told the real story — the gap between first and second place was 0.006, which is another way of saying it had no idea. The cause was our product copy, not the search. We'd never have found it on the demo path.

Ask to see the logs

Here's the one that catches people.

We had an AI chat feature that degraded gracefully when the provider failed — falls back to rule-based, shopper still gets a working store. Good design. It also meant that when the AI stopped working, the only signal was a slightly worse experience. No error. No alert. Nothing.

We spent an embarrassing amount of time guessing why. First theory: wrong model ID. Wrong. Second theory: an unsupported parameter. Also wrong — plausible, documented, and not what was happening.

Then we logged what the provider actually said:

status: 400
message: 'Your credit balance is too low to access the Anthropic API.'

The account was out of money. Not a code problem at all. Two rounds of confident guessing, and the answer was in a message we hadn't bothered to print.

So: ask a vendor what happens in their logs when the AI fails. If a graceful fallback means nobody finds out, they've built a system that can be broken for a week while everyone assumes it's fine. Graceful for the shopper, loud for the operator — that's the pair you want.

Ask what only a live run has taught them

This is the question that can't be faked, because the answer is a story.

Ours: we had an admin dropdown letting you pick the model — three options, sensible, tested. The chat code also passed an effort parameter to tune response depth. Both fine in isolation. Pick the wrong one of the three models and every request returned a 400, because that model doesn't accept that parameter.

We proved it rather than assuming, running the same request twice:

WITH effort     -> 400 "This model does not support the effort parameter."
WITHOUT effort  -> OK

No code review catches that. No test catches it unless you happen to test that combination. Only a live key against a live API finds it, and it sat there until we ran one.

Any vendor with a system actually in production has three of these stories and will tell them without much prompting. A vendor with a demo will describe their architecture instead.

Ask what it costs to run

Not the licence — the tokens.

If they can't tell you roughly what a thousand shopper conversations cost, one of two things is true: nobody's had a thousand conversations, or nobody's looked at the bill. Both are worth knowing before you sign.

The pattern

Notice what all four have in common: none of them are about the AI. They're about whether the thing has been run — by people who then had to figure out why it broke, pay for it, and fix it while someone waited.

The model is the easy part now. Everyone has access to the same ones. What differs is whether the system around it has met reality yet, and reality shows up as unrehearsed queries, silent failures, and a bill.

Ask for those. The demo will always work.