Before you let an AI agent touch your store: 5 questions to ask any vendor

The demo will be good. It's 2026 — the models are genuinely capable, and a vendor with a rehearsed script and a clean dataset can make almost anything look inevitable.

What the demo won't show you is what happens on a Tuesday when the agent misreads a message and does something to your store that you have to explain to your CFO. These five questions get at that. Ask them in this order.

1. Can it change anything, or only answer?

Start here, because everything else depends on it.

An agent that answers questions has one failure mode: it's wrong and you notice. An agent that acts — creates a promotion, edits a price, adjusts inventory, issues a refund — has a worse one: it's wrong, it's confident, and the first you hear is a customer using a discount nobody approved.

Plenty of "AI agents" are read-only, which is fine and much safer. But if a vendor is selling automation, they're selling write access. Make them say which.

2. Where is the boundary enforced — the prompt, or the code?

This is the question most vendors fail, and it's the one that matters.

If the answer involves the system prompt — "we instruct it to always ask first", "it's told never to exceed 20%" — that's not a boundary. It's a suggestion to a system whose entire job is interpreting language, from a vendor hoping nobody sends it a cleverly-worded message. Prompt instructions are the thing that fails under exactly the conditions you're worried about.

A real boundary lives somewhere the model doesn't get a vote. Ours is in the executor: research and action run through the same code path, and that path refuses anything with side effects while the agent is still investigating. The same promotion code is blocked while it looks around and runnable only after a human approves it. Nothing about that depends on the model behaving.

Ask: show me the line of code that stops it. If it's a sentence in English, it doesn't stop it.

3. What exactly am I approving?

"The agent will ask for confirmation" sounds reassuring until you see what the confirmation says. Often it's the agent's own summary of what it intends to do — which means you're approving a description written by the thing you're checking.

You want to see the actual change. When we asked our ops agent for a 10% promotion, it wrote the engine's own workflow code, and that code is what sits in the approval card:

import { createPromotionsWorkflow } from "@medusajs/medusa/core-flows"

export default async function({ container, log }: ExecArgs) {
  const { result } = await createPromotionsWorkflow(container).run({
    input: { promotionsData: [{ code: "MNTDEMO10", type: "standard",
      status: "active", application_method: { type: "percentage",
      target_type: "order", value: 10 } }] },
  })
}

You can read that and know precisely what will happen. "I'll create a 10% discount for you" is a claim about that code, and claims are what you're trying to verify.

4. Prove nothing happened before I clicked

Make them show you the database, not the interface.

A pending state in a UI is easy. A pending state that's actually pending is the whole engineering problem. When our proposal was sitting there awaiting approval, this is what the store looked like:

agent_action:  status=pending
promotion:     0 rows

The code existed. The promotion did not. After one click:

agent_action:  status=executed
               decided_by=user_01KXK2W4TYBXRS992DNM7EBNGJ
promotion:     MNTDEMO10 | active | percentage | value=10

That second block is the part to insist on. decided_by is a person — not "the agent did it", a named human, on the record. When something goes wrong at 2am you will want to know who approved it, and "the AI" is not an answer that survives a board meeting.

5. What happens when your AI provider goes down?

Not if. Providers rate-limit, have outages, change models, and expire billing. Ask what your store does that afternoon.

If the answer is a shrug, you've made your search — or worse, your checkout — dependent on someone else's uptime. The right answer is that every AI feature has a deterministic path underneath it and the store degrades instead of breaking.

We know ours does because we ran it that way for weeks: no API key, then a key with no credits, then a provider returning 400s. The store kept selling on rule-based chat and keyword search the whole time. Shoppers saw a worse experience. They didn't see an error.

What good answers sound like

A vendor worth buying from will answer question 2 with a mechanism, not a reassurance. They'll show you the diff rather than the summary. They'll open the database when you ask about the pending state. And they'll tell you what their thing can't do before you find out yourself.

Ours, for the record: our ops agent runs on our own platform, and the numbers above come from our own store with a demo catalog — not a client's live shop. We have no conversion data from it and don't claim any. What's tested is the part in question 4, and we'd rather you asked.