---
title: "Testing AI Agents for Consistent Results | Coding Delta"
description: "How we test Chatsome, our AI agent for customer service and sales, with an LLM test suite, so a change for one client never breaks answers for another."
canonical_url: "https://codingdelta.com/blog/testing-in-the-age-of-ai-agents"
last_updated: "2024-06-26T00:00:00.000Z"
meta:
  "og:description": "How we test Chatsome, our AI agent for customer service and sales, with an LLM test suite, so a change for one client never breaks answers for another."
  "og:image": "https://codingdelta.com/content/images/cover-testing-ai-agents.jpg"
  "og:image:alt": "Cover: 'Change one sentence. Test everything.' with a test summary going from 17 to 20 of 22 tests passing"
  "og:image:height": "900"
  "og:image:type": image/jpeg
  "og:image:width": "1600"
  "og:title": "Testing AI Agents for Consistent Results"
---

> Index of every page on this site: [llms.txt](https://codingdelta.com/llms.txt)

[Journal](https://codingdelta.com/blog)

AI agents26 June 2024·3 min read

# **Testing in the age of AI agents**

Using a testing framework is critical in creating consistent AI agents

![Cover: 'Change one sentence. Test everything.' with a test summary going from 17 to 20 of 22 tests passing](https://codingdelta.com/content/images/cover-testing-ai-agents.jpg)

Ask ChatGPT the same question twice and you get two different answers. That's fine in a chat window. It's a real problem when an AI agent talks to your customers and you need to know whether today's code change made it better or worse. This is how we test the AI agents behind [**Chatsome**](https://chatsome.co/).

## Key takeaways

- LLM output varies from run to run, so traditional tests that expect the same output for the same input fall short.
- We built an in-house test suite: each test defines a chat history and the expected output, and a separate GPT session judges whether the response matches.
- In one example, changing a single sentence in a prompt took us from 5 failing tests to 2.
- Without automated tests, you can't know whether a change actually improves your agent.

## Chatsome: an AI agent connected to your CRM

At Coding Delta we help clients grow, and we also build and market our own products. One of them is Chatsome, an AI chatbot that integrates with our clients' CRM systems.

Through that CRM integration, Chatsome becomes part of the company's workforce:

- **Digital sales agent:** the chatbot knows every product being sold and helps visitors 24/7.
- **Digital customer service agent:** with access to order information, it resolves many inquiries without involving a human customer service representative.

![Chatsome chat widget answering a visitor's question about door handles with product cards](https://codingdelta.com/content/images/chatsome-screenshot-widget.png)

## How our agents work

Website visitors ask all kinds of questions, from sales inquiries to customer service issues. We developed an extensive pipeline that determines the user's intent and the right response.

Our clients sell very different products, from door handles to baby accessories. So we built a generic workflow that adapts to any scenario, and our system handles all of them.

## The problem: testing something that never answers the same way twice

That variety creates a challenge: how do you test whether a code change improves or worsens the bot's performance for a specific client? With only a few clients, manual testing was feasible. As we scaled up and handled more edge cases, it became impractical.

On top of that, ChatGPT's responses vary every time you ask the same question. We asked "Who is the king of the Netherlands?" twice and got these answers:

> **Answer 1:** The current king of the Netherlands is King Willem-Alexander. He has been the monarch since April 30, 2013, following the abdication of his mother, Queen Beatrix.

> **Answer 2:** The king of the Netherlands is King Willem-Alexander. He has been the reigning monarch since April 30, 2013, following the abdication of his mother, Queen Beatrix. King Willem-Alexander is the first male monarch of the Netherlands in over a century, succeeding a line of three female monarchs.

The answers are similar, but different enough to be a problem when a computer has to interpret them.

## Our solution: an LLM test suite

Traditional testing assumes a function returns the same output for the same input. GPT doesn't, so we needed a different approach and developed an in-house test suite.

For every critical bot interaction, we write a test that specifies the chat history and the expected output. Then we:

1. Ask GPT to generate a response based on the chat history.
2. Start a new GPT session that compares the response with the expected output.
3. Score the result as "true" or "false".

This lets us verify every code change. If a test fails, we investigate and fix the issue until all tests pass.

## Why LLM tests are needed: one sentence, three more passing tests

Small changes can lead to very different outcomes. Below are two test runs where we changed only one prompt, and in that prompt only one sentence. The result: three more tests passed.

![Test suite run before the change: 22 tests, 17 passed, 5 failed](https://codingdelta.com/content/images/llm-test-5-failed.png)

![Test suite run after changing one sentence in the prompt: 22 tests, 20 passed, 2 failed](https://codingdelta.com/content/images/llm-test-2-failed.png)

This is why automated tests are essential when you work with LLMs. Without them, you never know whether a change to your code or prompts improves the final output.

## The impact

Since we introduced LLM testing, our development velocity has increased significantly. We are also more confident in the accuracy of the answers our AI agent gives our clients' visitors.

## Put AI agents to work for your business

Today, [**Chatsome**](https://chatsome.co/) runs AI agents for customer service and sales on your website, WhatsApp, email and voice. Want to use our AI agents, or learn more about our testing suite? Use the button below to get in touch. Looking at AI beyond customer service? See how we approach [**AI and automation**](https://codingdelta.com/automation).

## **Curious what this could do for you?**

We're happy to think along — over coffee on the Keizersgracht or on a video call.

```json
{"@context":"https://schema.org","@graph":[{"@id":"https://codingdelta.com/#website","@type":"WebSite","description":"Boutique software studio in Amsterdam for web apps, platforms, mobile apps and AI agents.","inLanguage":"en-US","name":"Coding Delta","url":"https://codingdelta.com/","publisher":{"@id":"https://codingdelta.com/#identity"},"workTranslation":{"@id":"https://codingdelta.com/nl#website"}},{"@id":"https://codingdelta.com/blog/testing-in-the-age-of-ai-agents#webpage","@type":"WebPage","dateModified":"2024-06-26T00:00:00.000Z","datePublished":"2024-06-26T00:00:00.000Z","description":"How we test Chatsome, our AI agent for customer service and sales, with an LLM test suite, so a change for one client never breaks answers for another.","name":"Testing AI Agents for Consistent Results","url":"https://codingdelta.com/blog/testing-in-the-age-of-ai-agents","breadcrumb":{"@id":"https://codingdelta.com/blog/testing-in-the-age-of-ai-agents#breadcrumb"},"isPartOf":{"@id":"https://codingdelta.com/#website"},"potentialAction":[{"@type":"ReadAction","target":["https://codingdelta.com/blog/testing-in-the-age-of-ai-agents"]}],"primaryImageOfPage":{"@type":"ImageObject","url":"https://codingdelta.com/content/images/cover-testing-ai-agents.jpg","inLanguage":"en-US","contentUrl":"https://codingdelta.com/content/images/cover-testing-ai-agents.jpg"}},{"@id":"https://codingdelta.com/#identity","description":"Boutique software studio in Amsterdam: strategy and scoping, product design, engineering (web apps, platforms, mobile) and AI & automation.","iso6523Code":"0106:77615999","legalName":"Coding Delta B.V.","name":"Coding Delta","url":"https://codingdelta.com","@type":["Organization","LocalBusiness","ProfessionalService"],"address":{"@type":"PostalAddress","streetAddress":"Keizersgracht 241","postalCode":"1016 EA","addressLocality":"Amsterdam","addressCountry":"NL"},"brand":[{"@type":"Brand","name":"Chatsome","url":"https://chatsome.co/","description":"AI agents for customer service and sales"},{"@type":"Brand","name":"Delta Security","url":"https://delta.codingdelta.com/","description":"Website security scans"},{"@type":"Brand","name":"RoundE","url":"https://rounde.nl/","description":"Employee participation for startups"}],"identifier":{"@type":"PropertyValue","propertyID":"KVK","value":"77615999"},"image":{"@id":"https://codingdelta.com/#/schema/image/1"},"knowsAbout":["Software development","Web application development","Product design","Mobile app development","AI agents","Business process automation"],"knowsLanguage":["en","nl"],"sameAs":["https://www.linkedin.com/company/codingdelta/"]},{"@id":"https://codingdelta.com/blog/testing-in-the-age-of-ai-agents#breadcrumb","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","name":"Home","item":"https://codingdelta.com","position":1},{"@type":"ListItem","name":"Blog","item":"https://codingdelta.com/blog","position":2},{"@type":"ListItem","name":"Testing in the age of AI agents","position":3}]},{"@id":"https://codingdelta.com/blog/testing-in-the-age-of-ai-agents#article","dateModified":"2024-06-26T00:00:00.000Z","datePublished":"2024-06-26T00:00:00.000Z","description":"How we test Chatsome, our AI agent for customer service and sales, with an LLM test suite, so a change for one client never breaks answers for another.","headline":"Testing in the age of AI agents","inLanguage":"en-US","thumbnailUrl":"https://codingdelta.com/content/images/cover-testing-ai-agents.jpg","@type":["Article","BlogPosting"],"articleSection":["AI agents"],"author":{"@id":"https://codingdelta.com/#identity"},"image":["https://codingdelta.com/content/images/cover-testing-ai-agents.jpg"],"isPartOf":{"@id":"https://codingdelta.com/blog/testing-in-the-age-of-ai-agents#webpage"},"keywords":["AI agents"],"mainEntityOfPage":{"@id":"https://codingdelta.com/blog/testing-in-the-age-of-ai-agents#webpage"},"publisher":{"@id":"https://codingdelta.com/#identity"}},{"@id":"https://codingdelta.com/#/schema/image/1","@type":"ImageObject","contentUrl":"https://codingdelta.com/og-image.png","inLanguage":"en-US","url":"https://codingdelta.com/og-image.png"},{"@id":"https://codingdelta.com/#logo","@type":"ImageObject","caption":"Coding Delta","contentUrl":"https://codingdelta.com/logo.png","inLanguage":"en-US","url":"https://codingdelta.com/logo.png"},{"@id":"https://codingdelta.com/#organization","@type":"Organization","logo":"https://codingdelta.com/logo.png","name":"Coding Delta","url":"https://codingdelta.com","address":{"@type":"PostalAddress","streetAddress":"Keizersgracht 241","postalCode":"1016 EA","addressLocality":"Amsterdam","addressCountry":"NL"},"sameAs":["https://www.linkedin.com/company/codingdelta/"]}]}
```

## Sitemap

See the full [sitemap](https://codingdelta.com/sitemap.md) for all pages.
