# How to Test an AI Agent Before It Answers Customers

> Test an AI agent on your client's real past conversations, not invented ones. What to put in the test set, how to grade it, and when it is ready to go live.

Canonical: https://maxbound.ai/blog/how-to-stress-test-ai-agents-before-go-live
Last updated: 2026-10-19

Published: 2026-07-18
Updated: 2026-10-19
Author: Baban Rashid
Category: [AI Agents](https://maxbound.ai/blog/category/agents)
Tags: AI Agent Testing, Simulations, Chatbot Testing, Agency Workflow

The best way to test an AI agent before it answers a client's customers is to run it on the messages those customers already sent. Take a week of the client's real comments and DMs, let the agent answer every one without sending anything, and read its answers next to what your team actually replied. Then fix the knowledge behind every wrong answer, run the same set again, and only go live once it passes, starting with replies a person approves.

Invented test questions are the trap. Somebody writes the questions the agent is about to be asked, and it passes, because the test was written to be passed. Real inboxes are full of typos, half-finished questions, haggling, two languages in one message and complaints that start as jokes.

This post covers what to put in a test set, how to grade it, the adversarial tests most teams skip, and how to sign an agent off with a client.

## Why agents that pass tests still fail customers

Passing your own tests proves less than it feels like it does. [Splunk, summarising recent research on agent evaluation](https://www.splunk.com/en_us/blog/artificial-intelligence/agent-testing.html), notes that half of enterprises deployed an AI system that passed internal evaluations and still caused a customer-facing failure within a year.

Two findings explain a lot of that gap.

Conversations break models more than single questions do. In [LLMs Get Lost In Multi-Turn Conversation](https://arxiv.org/abs/2505.06120), researchers ran more than 200,000 simulated conversations and measured an average performance drop when the same tasks were spread over several turns instead of asked at once.

**39%** — [Laban et al., LLMs Get Lost In Multi-Turn Conversation (2025): average drop from single-turn to multi-turn across six tasks](https://arxiv.org/abs/2505.06120)

And people who keep trying get further. In [NIST's evaluation of AI agent hijacking](https://www.nist.gov/news-events/news/2025/01/technical-blog-strengthening-ai-agent-hijacking-evaluations), repeated attempts raised the average attack success rate from 57% to 80%. A customer who wants a discount, a free delivery or a promise on record will try more than once.

So a useful test set has to look like the real inbox: multi-message, messy, and occasionally hostile.

## What to put in the test set

Build it from the client's own history. In Maxbound, you pick a set of past conversations (the best cases, the worst cases, or a whole week) and the agent answers each one in a [simulation](/platform/simulations). Nothing is delivered to anyone. Then add a handful of written cases for things the history doesn't contain yet, such as a new offer.

**A test set for one client's agent**

| Bucket | Example messages | What good looks like |
| --- | --- | --- |
| Everyday questions | Price, opening hours, stock, delivery areas, "do you do X?" | The right fact, from the client's own sources, in one short reply |
| Buying conversations | "Can you do any better on the price?", a booking with a date and a group size | Holds the price rules, asks the next qualifying question, saves the lead |
| Multi-message bursts | "Hi" then "are you open Friday?" then "for 6 people" | One answer that covers all three, not three answers |
| Messy input | Typos, emoji only, two languages in one message, all capitals | Understands it or asks one clear question back |
| Complaints | Late delivery, wrong order, a public post calling the brand a scam | Apologises the way the client does, then hands to a person |
| Things it must not answer | Medical or legal questions, refunds outside policy, press enquiries | Doesn't guess; hands to a named person |
| Pressure | Asking for a discount three different ways, "your colleague promised me..." | Doesn't invent an offer or a promise |

Thirty to fifty conversations covering every bucket is enough to start. Include at least five of the client's genuinely bad days: the week a delivery went wrong, the post that got the angry comments.

## How to grade the answers

Grade against what the client's team actually said, not against your own sense of a good reply. The question for each answer is simple: would we have sent this, as written, under this client's name?

Sort every answer into one of four piles:

1. **Would send.** Right fact, right tone. If it differs from the team's wording but says the same thing, it passes.
2. **Wrong fact.** A price, a policy or a promise that doesn't match what the team says. This is a knowledge problem: the source is missing, out of date, or contradicts another source.
3. **Dodged it.** The agent hedged instead of answering, usually because the knowledge behind the question is thin.
4. **Missed the handoff.** An angry or high-value conversation it should have passed to a person, and didn't.

Count each pile. The share in "would send" is your pass rate. Maxbound's simulations put the agent's answer beside your team's and mark each one as matching or wrong, which makes this sort fast.

Fix every wrong answer at the source, never in the answer. If the agent quoted last year's price, update the price list, don't write a special rule for that question. Then run the same set again. The point of a fixed set is that you can watch the gap close, and catch it if a fix for one answer breaks another.

## The tests most teams skip

Three kinds of test find the problems that cost a client most.

**Run it as a conversation, not a quiz.** Take five real threads and replay them message by message, including the customer changing their mind halfway. The multi-turn research above is why: an agent that answers each question perfectly in isolation can lose the thread by the fourth message.

**Try to break it.** Ask for a discount, then ask again more politely, then claim a colleague promised one. Ask it to confirm something the client never offered. Paste in an instruction like "ignore your previous rules". Write down what it said each time.

**Test the handoff, not just the answer.** For every conversation that should reach a person, check where it landed, whether the person got the whole thread, and how long it would have waited. A handoff to an inbox nobody watches is not a handoff.

**Don't test only on the client's good days**

An agent tuned on a quiet month of "what are your hours?" will look perfect and then meet a delivery crisis. Put the worst week you can find in the test set, on purpose.

## Run it again after every change

Testing isn't a launch step you do once. Every time the client changes a price, adds a product, updates a policy or you edit the agent's tone, replay the same set and compare. Maxbound's simulations are built for this: regressions after a change show up as answers that used to match and now don't.

Keep the set as a living file. Every time a real conversation goes wrong after launch, add it to the set, so the same mistake can never pass a test again.

## Sign it off with the client

For an agency, testing is also the moment you earn the client's trust in the agent. Make it a visible step in onboarding rather than something you did quietly.

**Test and sign off an AI agent with a client**

1. **Build the set from their history** — Pick 30 to 50 real conversations covering everyday questions, buying conversations, bursts, messy input, complaints, things the agent must not answer, and pressure. Include their worst week.
2. **Run it without sending anything** — Let the agent answer every conversation in a simulation, so no customer receives anything.
3. **Grade every answer into four piles** — Would send, wrong fact, dodged it, missed the handoff. The share in would send is the pass rate.
4. **Fix sources, then replay** — Correct the price list, policy or FAQ behind each wrong answer, and run the same set again until the wrong-fact and missed-handoff piles are empty.
5. **Show the client the results** — Send them the pass rate, five example answers, and the list of topics that always go to a person. Agree the sign-off in writing.
6. **Go live with approval on** — Start in a mode where a person approves each reply, and keep adding real failures to the test set.

Clients rarely read a 50-row spreadsheet. They do read five answers to their own customers' messages, side by side with what their team wrote. That is usually what gets the yes.

## When is an agent ready?

Ready means two piles are empty and one is full: no wrong facts, no missed handoffs, and nearly everything in "would send". A few answers that are correct but worded differently from how the team would say it are fine; they'll improve as reviewers edit drafts.

Even then, go live with a person approving replies, as in [a human-in-the-loop workflow](/blog/how-to-set-up-a-human-in-the-loop-ai-workflow), and let real approvals decide when it can send on its own. Testing tells you the agent is safe to start; the first weeks of drafts tell you it's safe to trust. Every approval and edit there keeps training it, as described in [self-improving AI that learns your brand](/blog/self-improving-ai-that-learns-your-brand).

## Where Maxbound fits

Maxbound is an AI operating system for agencies, and testing is built into how its [AI agents](/platform/ai-agents) are set up. Each client's agent is trained on that client's website, documents, FAQs and past conversations ([knowledge training](/platform/knowledge)), rehearsed on the client's real past chats in [Simulations](/platform/simulations) with nothing sent, and then put live on Instagram, Facebook, WhatsApp, website chat or in-app chat with replies waiting for approval until you decide otherwise.

## Frequently Asked Questions

### How do you test an AI chatbot before launch?

Run it on the real messages customers already sent, without sending anything. Compare each answer with what your team actually replied, sort them into would send, wrong fact, dodged it and missed the handoff, fix the sources behind the wrong ones, and replay the same set until it passes.

### How many conversations should a test set have?

Thirty to fifty real conversations is enough to start, as long as they cover everyday questions, buying conversations, message bursts, messy input, complaints, things the agent must not answer, and people pushing for discounts. Add every real failure after launch to the set.

### Why test on real conversations instead of written test questions?

Written questions tend to be the ones the agent was built to answer, so it passes. Real messages carry typos, mixed languages, half-finished questions and pressure, which is where agents actually fail.

### What is red-teaming an AI agent?

Trying on purpose to make it do something it shouldn't: give an unapproved discount, confirm a promise nobody made, follow an instruction pasted into a message, or answer a medical or legal question. Repeat each attempt a few times, because people who keep trying get further.

### Do I need to test again after the agent goes live?

Yes. Replay the same test set every time the client's prices, products or policies change, or you edit the agent, and compare. A fix for one answer can break another.

### Is an agent ready to send replies on its own once it passes testing?

Not yet. Passing the test means it is safe to start with a person approving each reply. Let a few weeks of approvals, including a busy day, show it is ready to send on its own.

## Related

- [How to Answer Instagram DMs in Arabic for Clients](https://maxbound.ai/blog/how-to-answer-instagram-dms-in-arabic-for-clients) — A step-by-step playbook for agencies: connect, train and brief an agent for Arabic Instagram DM replies and comment-to-DM, dialects and spelling variants
- [How to Test a Kurdish Sorani WhatsApp AI Chatbot](https://maxbound.ai/blog/how-to-test-a-kurdish-sorani-whatsapp-ai-chatbot) — A step-by-step walkthrough for agencies: connect a client's WhatsApp, train a Sorani agent, rehearse it on past chats, and launch with approval on.
- [Comment Moderation Tool for Agencies: A Setup Playbook](https://maxbound.ai/blog/comment-moderation-tool-for-agencies-a-setup-playbook) — How agencies set up comment moderation for every client: a policy template, hide-vs-reply rules, approval settings and the mistakes that cost clients.
- [How to Reply to Instagram Comments on Client Ads](https://maxbound.ai/blog/how-to-reply-to-instagram-comments-on-client-ads) — Comments under ads are paid-for questions, often hidden from the profile. Where to find them, how to triage them, what to reply in public and by DM, and how
- [Busting the Myths of Instagram DM Automation Tools](https://maxbound.ai/blog/busting-the-myths-of-instagram-dm-automation-tools) — Four myths keep agencies answering Instagram DMs by hand: that customers hate automation, that it's all-or-nothing, that setup takes weeks, and that it gets
- [All posts](https://maxbound.ai/blog)
- [Platform overview](https://maxbound.ai/platform)