[{"data":1,"prerenderedAt":239},["ShallowReactive",2],{"post-\u002Fen\u002Fblog\u002Ftesting-in-the-age-of-ai-agents":3,"post-surround-\u002Fen\u002Fblog\u002Ftesting-in-the-age-of-ai-agents":228},{"id":4,"title":5,"authors":6,"badge":7,"body":9,"date":211,"dateModified":6,"description":212,"extension":213,"image":214,"meta":219,"navigation":220,"path":221,"seo":222,"stem":225,"translation":226,"__hash__":227},"blog\u002Fen\u002Fblog\u002Ftesting-in-the-age-of-ai-agents.md","Testing in the age of AI agents",null,{"label":8},"AI agents",{"type":10,"value":11,"toc":199},"minimark",[12,24,29,45,49,52,55,70,77,81,84,87,91,94,97,106,114,117,121,124,127,139,142,146,149,173,176,180,183,187],[13,14,15,16,23],"p",{},"Ask ChatGPT the same question twice and you get two different answers. That's fine in a chat window. It's a real problem when an AI agent talks to your customers and you need to know whether today's code change made it better or worse. This is how we test the AI agents behind ",[17,18,22],"a",{"href":19,"rel":20,"target":21},"https:\u002F\u002Fchatsome.co\u002F",[],"_blank","Chatsome",".",[25,26,28],"h2",{"id":27},"key-takeaways","Key takeaways",[30,31,32,36,39,42],"ul",{},[33,34,35],"li",{},"LLM output varies from run to run, so traditional tests that expect the same output for the same input fall short.",[33,37,38],{},"We built an in-house test suite: each test defines a chat history and the expected output, and a separate GPT session judges whether the response matches.",[33,40,41],{},"In one example, changing a single sentence in a prompt took us from 5 failing tests to 2.",[33,43,44],{},"Without automated tests, you can't know whether a change actually improves your agent.",[25,46,48],{"id":47},"chatsome-an-ai-agent-connected-to-your-crm","Chatsome: an AI agent connected to your CRM",[13,50,51],{},"At Coding Delta we help clients grow, and we also build and market our own products. One of them is Chatsome, an AI chatbot that integrates with our clients' CRM systems.",[13,53,54],{},"Through that CRM integration, Chatsome becomes part of the company's workforce:",[30,56,57,64],{},[33,58,59,63],{},[60,61,62],"strong",{},"Digital sales agent:"," the chatbot knows every product being sold and helps visitors 24\u002F7.",[33,65,66,69],{},[60,67,68],{},"Digital customer service agent:"," with access to order information, it resolves many inquiries without involving a human customer service representative.",[13,71,72],{},[73,74],"img",{"alt":75,"src":76},"Chatsome chat widget answering a visitor's question about door handles with product cards","\u002Fcontent\u002Fimages\u002Fchatsome-screenshot-widget.png",[25,78,80],{"id":79},"how-our-agents-work","How our agents work",[13,82,83],{},"Website visitors ask all kinds of questions, from sales inquiries to customer service issues. We developed an extensive pipeline that determines the user's intent and the right response.",[13,85,86],{},"Our clients sell very different products, from door handles to baby accessories. So we built a generic workflow that adapts to any scenario, and our system handles all of them.",[25,88,90],{"id":89},"the-problem-testing-something-that-never-answers-the-same-way-twice","The problem: testing something that never answers the same way twice",[13,92,93],{},"That variety creates a challenge: how do you test whether a code change improves or worsens the bot's performance for a specific client? With only a few clients, manual testing was feasible. As we scaled up and handled more edge cases, it became impractical.",[13,95,96],{},"On top of that, ChatGPT's responses vary every time you ask the same question. We asked \"Who is the king of the Netherlands?\" twice and got these answers:",[98,99,100],"blockquote",{},[13,101,102,105],{},[60,103,104],{},"Answer 1:"," The current king of the Netherlands is King Willem-Alexander. He has been the monarch since April 30, 2013, following the abdication of his mother, Queen Beatrix.",[98,107,108],{},[13,109,110,113],{},[60,111,112],{},"Answer 2:"," The king of the Netherlands is King Willem-Alexander. He has been the reigning monarch since April 30, 2013, following the abdication of his mother, Queen Beatrix. King Willem-Alexander is the first male monarch of the Netherlands in over a century, succeeding a line of three female monarchs.",[13,115,116],{},"The answers are similar, but different enough to be a problem when a computer has to interpret them.",[25,118,120],{"id":119},"our-solution-an-llm-test-suite","Our solution: an LLM test suite",[13,122,123],{},"Traditional testing assumes a function returns the same output for the same input. GPT doesn't, so we needed a different approach and developed an in-house test suite.",[13,125,126],{},"For every critical bot interaction, we write a test that specifies the chat history and the expected output. Then we:",[128,129,130,133,136],"ol",{},[33,131,132],{},"Ask GPT to generate a response based on the chat history.",[33,134,135],{},"Start a new GPT session that compares the response with the expected output.",[33,137,138],{},"Score the result as \"true\" or \"false\".",[13,140,141],{},"This lets us verify every code change. If a test fails, we investigate and fix the issue until all tests pass.",[25,143,145],{"id":144},"why-llm-tests-are-needed-one-sentence-three-more-passing-tests","Why LLM tests are needed: one sentence, three more passing tests",[13,147,148],{},"Small changes can lead to very different outcomes. Below are two test runs where we changed only one prompt, and in that prompt only one sentence. The result: three more tests passed.",[150,151,157,165],"div",{"className":152},[153,154,155,156],"flex","flex-col","sm:flex-row","sm:gap-3",[150,158,159],{},[13,160,161],{},[73,162],{"alt":163,"src":164},"Test suite run before the change: 22 tests, 17 passed, 5 failed","\u002Fcontent\u002Fimages\u002Fllm-test-5-failed.png",[150,166,167],{},[13,168,169],{},[73,170],{"alt":171,"src":172},"Test suite run after changing one sentence in the prompt: 22 tests, 20 passed, 2 failed","\u002Fcontent\u002Fimages\u002Fllm-test-2-failed.png",[13,174,175],{},"This is why automated tests are essential when you work with LLMs. Without them, you never know whether a change to your code or prompts improves the final output.",[25,177,179],{"id":178},"the-impact","The impact",[13,181,182],{},"Since we introduced LLM testing, our development velocity has increased significantly. We are also more confident in the accuracy of the answers our AI agent gives our clients' visitors.",[25,184,186],{"id":185},"put-ai-agents-to-work-for-your-business","Put AI agents to work for your business",[13,188,189,190,193,194,23],{},"Today, ",[17,191,22],{"href":19,"rel":192,"target":21},[]," runs AI agents for customer service and sales on your website, WhatsApp, email and voice. Want to use our AI agents, or learn more about our testing suite? Use the button below to get in touch. Looking at AI beyond customer service? See how we approach ",[17,195,198],{"href":196,"rel":197},"\u002Fautomation",[],"AI and automation",{"title":200,"searchDepth":201,"depth":201,"links":202},"",2,[203,204,205,206,207,208,209,210],{"id":27,"depth":201,"text":28},{"id":47,"depth":201,"text":48},{"id":79,"depth":201,"text":80},{"id":89,"depth":201,"text":90},{"id":119,"depth":201,"text":120},{"id":144,"depth":201,"text":145},{"id":178,"depth":201,"text":179},{"id":185,"depth":201,"text":186},"2024-06-26","Using a testing framework is critical in creating consistent AI agents","md",{"src":215,"alt":216,"height":217,"width":218},"\u002Fcontent\u002Fimages\u002Fcover-testing-ai-agents.jpg","Cover: 'Change one sentence. Test everything.' with a test summary going from 17 to 20 of 22 tests passing",900,1600,{},true,"\u002Fen\u002Fblog\u002Ftesting-in-the-age-of-ai-agents",{"title":223,"description":224},"Testing AI Agents for Consistent Results","How we test Chatsome, our AI agent for customer service and sales, with an LLM test suite, so a change for one client never breaks answers for another.","en\u002Fblog\u002Ftesting-in-the-age-of-ai-agents","testen-in-het-tijdperk-van-ai-agenten","SgO-TjyRjEE8TXvxn3HWE_Zw3XYP4coiH5Hpi3J2Bvo",[229,234],{"title":230,"path":231,"stem":232,"description":233,"children":-1},"Optimizing Air Quality Monitoring with IoT: A Case Study","\u002Fblog\u002Foptimizing-air-quality-monitoring-with-iot","en\u002Fblog\u002Foptimizing-air-quality-monitoring-with-iot","How we use 100s of sensors to improve air quality.",{"title":235,"path":236,"stem":237,"description":238,"children":-1},"How Familiebank is Reforming Family Loans: An Interview with Josca Buitenhuis","\u002Fblog\u002Fhow-familiebank-is-reforming-family-loans-interview-with-josca-buitenhuis","en\u002Fblog\u002Fhow-familiebank-is-reforming-family-loans-interview-with-josca-buitenhuis","Discover how Josca Buitenhuis simplifies family loans with Familiebank. Read about her vision, plans, and collaboration with Coding Delta.",1795282281241]