Test the Fix Without Reintroducing Yesterday's AI Workflow Bug

On Monday morning, Casey fixes the rush charge. Casey handles order operations at Fernway Supply, a fictional promo distributor, and for two weeks the AI order workflow has been leaving the rush fee off orders where the customer wrote "need by Friday" instead of ticking the rush box. Casey adds a clear rule to the instructions, runs the three rush orders that failed, and all three come out right. There is a small celebration involving the good biscuits.

On Tuesday afternoon, a customer's partial shipment order arrives: 200 tote bags now, 300 later, both with the same in hands date. The workflow, newly enthusiastic about dates, reads "same date, two shipments" as a rush and adds the fee to both. This order type had worked fine for months. Nobody tested it on Monday, because nobody was thinking about partial shipments on Monday. They were thinking about biscuits.

Before releasing a fix to an AI workflow, test the original failure and a set of cases that already worked. Check critical fields and safe stops separately from wording. Then confirm the finished record in the destination system, because correct extraction alone doesn't prove the order or invoice landed correctly.

The tool for this is small and unglamorous: a test register. It's a list of approved inputs, what the right result is for each, and what happened the last time you ran them. It grows every time a real problem shows you a gap.

What goes in the register

A useful register has three kinds of cases. Ordinary work that should keep working. Awkward cases that test specific decisions. And every past failure, kept as a reminder. For an order workflow like Casey's, the starting register might look like this:

Notice the cases that are supposed to stop. A workflow that always returns a confident answer can look successful while making decisions it has no basis for. For each safe stop, write down the question it should ask and what information should come with it.

Save the original failure exactly as it arrived

When something breaks, save an approved copy of the input that caused it, the output it produced, and the correct result. Remove private information you don't need before the case goes into a shared test set.

Don't tidy the example up. The odd formatting, the note in the wrong field, or the attachments in an unusual order may be exactly why it failed. A cleaned version can pass while the real one still breaks.

Describe the failure in business terms. "Wrong result" won't help anyone next month. "Added a rush fee to both shipments of a partial order with matching dates" tells the next person what to check. Record the workflow version and the time of the run too. A screenshot helps explain the problem, but keep the actual record and run history if you have them.

Separate critical failures from cosmetic ones

A reworded sentence in an internal note may be fine. A changed quantity, the wrong customer account, a missing approval, or a promise to a customer is not.

Decide up front which fields need exact checks and which outputs need a person's judgment. Dates, totals, IDs, and which actions were allowed can usually be checked exactly. Whether a summary is useful usually needs a reviewer.

Keep the scoring visible. One overall percentage can hide a serious error behind dozens of correct minor fields, so report critical results on their own. And mark anything you didn't run as "not tested," never as a pass. Missing evidence is not good news.

Decide what counts as passing before you look

Write the release criteria before you see the new results. Once a revised workflow looks better in most ways, it's tempting to explain away the one failure that matters.

A reasonable gate for Casey's team might be: every critical field correct on every case, every safe stop still stops, and no new differences on ordinary orders that nobody can explain. The exact gate depends on what a mistake costs. What matters is that it's written down first and that one person owns the call.

Run the same cases, under the same conditions

Run the same cases before and after the change. Keep the source data and settings stable, and record anything that changed underneath you.

A model update, a connector change, or a change in the source system can alter behavior even when nobody touched the instructions. Write down enough about each run to tell those apart from the change you meant to test.

Some AI outputs vary from run to run. For consequential decisions, run important cases more than once and check that the decision stays inside the approved boundary each time. One correct answer is weak evidence if the same input produces a different decision on the next try.

Use a safe destination for any test that writes. A test should never create a real order or send a customer message unless that environment and that action have been explicitly approved.

Check the whole path, not just the decision

A corrected extraction can still break a later step if it changes a field format or drops a value the next system needs. After testing the specific decision, run the workflow all the way to its approved finish line.

These checks catch failures that a test of the instructions alone can't see. A workflow that passes every case but isn't listening to the inbox anymore has a perfect score and no output. For retries, AWS's guidance on making retries safe with idempotent APIs is a useful reference for recognizing a repeated request without repeating its effect.

Release it as a controlled change

Give each change an owner, a version, and a short description. Record which register cases were run, which passed, anything still known to be wrong, and the condition that would make you pause or roll back.

Avoid bundling unrelated fixes into one release. Smaller releases make it much easier to connect a surprise with the change that caused it.

Start with a bounded slice of real work and look at the results. Keep the manual fallback ready until you've seen the conditions that matter. If you're also deciding how much the workflow can do without approval, that's a separate decision, covered in our rollout checklist.

Keep the register alive

When a new problem shows up, add it to the register after you fix it. Don't just add another sentence to the instructions and hope the growing paragraph covers everything. The register is what remembers.

Assign someone to maintain it as the business changes. New suppliers, pricing policies, document formats, and approval rules should show up as new cases. Retire old cases carefully, and keep a note of why a rule changed so an outdated case doesn't later get mistaken for a current requirement.

If your workflow is split into stages, keep a few cases per stage as well as the end to end ones. Our post on small, single purpose agents covers how stage boundaries make it easier to see where a failure started.

Release checklist

• The original failure is saved as it arrived, with the correct result written down.
• The register includes ordinary cases, awkward cases, safe stops, and every past failure.
• Critical fields are checked exactly and reported separately from cosmetic differences.
• Pass and fail criteria were written before the new results were seen.
• Old and new versions ran on the same cases under recorded conditions.
• Important decisions were run more than once.
• Write tests used a safe destination.
• The destination record, triggers, permissions, and any restarted listener were checked after the change.
• The change has an owner, a version, and a pause or rollback condition.
• New failures go into the register after they're fixed.

Frequently asked questions

1What is a regression test for an AI workflow?

It's a rerun of cases the workflow already handled correctly, done after a change, to confirm the change didn't break them. For AI workflows, that usually means a saved set of real inputs with known correct results, including cases that should stop and ask a person.

2How many test cases does an AI workflow need?

Enough to cover the different decisions the workflow makes and every failure you've already seen, rather than a large pile of similar clean orders. There's no universal number. The register should grow as real problems reveal gaps.

3Why does an AI workflow give different results on the same input?

Some AI outputs vary between runs, and a model, connector, or source system can change without anyone editing the instructions. For important decisions, run cases more than once and record what changed between runs.

4Is it enough to test the AI's output without checking the destination system?

No. A correct extraction can still produce a wrong record if a field format changes or a required value goes missing on the way. Check the finished record in a safe environment, along with whatever triggers the workflow.

5Who should approve releasing a change to an AI workflow?

One named owner, using criteria written before the new results were seen. That person decides whether the change ships, what slice of work it starts on, and what would cause a pause or rollback.

What the register can't promise

A test register can't prove how the workflow will behave on every future input. It shows that the tested cases still work. Repeated runs, a gradual release, and a close eye on critical failures are still needed.

Wednesday at Fernway

On Wednesday, Casey adds two rows to the register: the partial shipment order from Tuesday and a second partial order with different dates, just to be thorough. The fix this time is narrower. Before releasing it, Casey runs all nine cases, including both rush cases from Monday and the two that should stop. Everything passes, the tote bag order comes out as two shipments with no rush fee, and the inbox listener is confirmed running afterward.

There are no biscuits on Wednesday. There is, however, a register with nine rows that remembers Monday and Tuesday, so that next month Casey doesn't have to.