Ask ChatGPT the same question twice and you get two different answers. That's fine in a chat window. It's a real problem when an AI agent talks to your customers and you need to know whether today's code change made it better or worse. This is how we test the AI agents behind Chatsome.

Key takeaways

  • LLM output varies from run to run, so traditional tests that expect the same output for the same input fall short.
  • We built an in-house test suite: each test defines a chat history and the expected output, and a separate GPT session judges whether the response matches.
  • In one example, changing a single sentence in a prompt took us from 5 failing tests to 2.
  • Without automated tests, you can't know whether a change actually improves your agent.

Chatsome: an AI agent connected to your CRM

At Coding Delta we help clients grow, and we also build and market our own products. One of them is Chatsome, an AI chatbot that integrates with our clients' CRM systems.

Through that CRM integration, Chatsome becomes part of the company's workforce:

  • Digital sales agent: the chatbot knows every product being sold and helps visitors 24/7.
  • Digital customer service agent: with access to order information, it resolves many inquiries without involving a human customer service representative.

How our agents work

Website visitors ask all kinds of questions, from sales inquiries to customer service issues. We developed an extensive pipeline that determines the user's intent and the right response.

Our clients sell very different products, from door handles to baby accessories. So we built a generic workflow that adapts to any scenario, and our system handles all of them.

The problem: testing something that never answers the same way twice

That variety creates a challenge: how do you test whether a code change improves or worsens the bot's performance for a specific client? With only a few clients, manual testing was feasible. As we scaled up and handled more edge cases, it became impractical.

On top of that, ChatGPT's responses vary every time you ask the same question. We asked "Who is the king of the Netherlands?" twice and got these answers:

Answer 1: The current king of the Netherlands is King Willem-Alexander. He has been the monarch since April 30, 2013, following the abdication of his mother, Queen Beatrix.

Answer 2: The king of the Netherlands is King Willem-Alexander. He has been the reigning monarch since April 30, 2013, following the abdication of his mother, Queen Beatrix. King Willem-Alexander is the first male monarch of the Netherlands in over a century, succeeding a line of three female monarchs.

The answers are similar, but different enough to be a problem when a computer has to interpret them.

Our solution: an LLM test suite

Traditional testing assumes a function returns the same output for the same input. GPT doesn't, so we needed a different approach and developed an in-house test suite.

For every critical bot interaction, we write a test that specifies the chat history and the expected output. Then we:

  1. Ask GPT to generate a response based on the chat history.
  2. Start a new GPT session that compares the response with the expected output.
  3. Score the result as "true" or "false".

This lets us verify every code change. If a test fails, we investigate and fix the issue until all tests pass.

Why LLM tests are needed: one sentence, three more passing tests

Small changes can lead to very different outcomes. Below are two test runs where we changed only one prompt, and in that prompt only one sentence. The result: three more tests passed.

This is why automated tests are essential when you work with LLMs. Without them, you never know whether a change to your code or prompts improves the final output.

The impact

Since we introduced LLM testing, our development velocity has increased significantly. We are also more confident in the accuracy of the answers our AI agent gives our clients' visitors.

Put AI agents to work for your business

Today, Chatsome runs AI agents for customer service and sales on your website, WhatsApp, email and voice. Want to use our AI agents, or learn more about our testing suite? Use the button below to get in touch. Looking at AI beyond customer service? See how we approach AI and automation.

Curious what this could do for you?

We're happy to think along — over coffee on the Keizersgracht or on a video call.