Artificial Intelligence, zBlog
Why Your Multi-Agent LLM System Needs an LLM-as-Judge Eval, and What Building One Actually Looks Like
trantorindia | Updated: August 28, 2026
If you’re running an AI system made up of multiple specialized “agents” working together, you probably have some tests in place. But here’s an uncomfortable question: do you actually know if your system is making good decisions, not just technically working ones?
We didn’t either. So we built a way to find out, and it changed how we thought about our own system.
The Problem
Our AI assistant has eight specialist “agents.” One agent figures out which specialist should handle a question, that specialist picks the right tool to use, then pulls out the details it needs to answer.
We had almost 400 tests checking this routing, but they only used simple, obvious phrasing. What happens when someone asks the same question in a roundabout way? What if a request actually needs two specialists working together? What about someone saying “next weekend” instead of a specific date?
None of that was being tested. Our system could be quietly sending questions to the wrong specialist over and over, and we’d only find out when users complained, not from our test results.
The real danger with AI systems isn’t that they crash. It’s that they give a confidently wrong answer that looks correct.
Our Solution: Let an AI Grade the AI
We considered other options first. Simple rule-based tests are great for clear-cut cases but can’t judge “was this a reasonable call?” Human reviewers are accurate but can’t check hundreds of decisions every time we make a change. Testing with real users works, but only after we’ve already shipped something risky.
So we used a second AI as a judge: it looks at what our system decided and scores whether that decision made sense. It’s not perfect, and it costs extra to run, but it’s far cheaper than shipping a bug that quietly sends half our users to the wrong place.
How We Built It
We put together a simple system with a few moving parts: test questions, the real decision-making code (not a fake stand-in), a “judge” AI with instructions on how to score each kind of decision, and a way to log the results and how long each step took.
Making Sure the Judge Could Be Trusted
Before trusting any of the scores, we tried to catch the judge making mistakes. We ran the same test questions through judge models from two different companies and got identical scores every time. That’s when we started trusting the results: if two unrelated AIs agree, the problem is really in our system, not in one judge’s quirks.
We were careful about one thing especially: we never tweaked the judge’s instructions just to make our scores look better. If a score dropped after a code change, that meant something in the code needed fixing, not the test.
Three False Alarms We Had to Untangle
At first, a few parts of our system scored a flat zero, but for three completely different reasons that all looked the same on the surface:
Bad test data.
Some of our “correct answers” for tests were guessed, not based on the actual code. Once we fixed the expected answers, the score jumped back up.
Mixed-up tests.
Test questions meant for one specialist were accidentally being graded against a different specialist. Fixing that instantly fixed five “failing” scores.
A missing setting.
One test failed simply because a key wasn’t configured, nothing to do with the actual AI logic.
The lesson: a zero score can mean three totally different things. You have to look at individual results, not just the summary number, to know which one you’re dealing with.
What We Actually Found
Once the tests were solid, we found real problems:
Misrouted questions.
Three out of five reworded questions for one specialist went to the wrong specialist entirely, because they didn’t use the exact keywords our system was expecting.
A hidden failure mode.
When the system couldn’t properly read its own routing decision, it silently defaulted to some other specialist instead of flagging an error. Users still got an answer, just from the wrong place, with no indication anything went wrong.
A design flaw, not just a bug.
Our date-understanding feature worked for “in 2 weeks” but failed on “in two weeks,” because it was built pattern-by-pattern instead of actually understanding language. That’s not a quick fix, it’s a sign the approach needs to be rethought.
What We’d Tell Other Teams
The very first decision your system makes matters most. Everything downstream depends on it, so test it with tricky, indirect phrasing, not just the obvious stuff.
Made-up test questions are a starting point, not the finish line. Real user questions, over time, are what actually tell you the truth.
You need both kinds of testing. Simple rule-based tests catch clear-cut breakages. AI-judged tests catch “this technically works but is subtly wrong.” Neither one replaces the other.
The Bottom Line
This process caught three real problems we had no idea existed: a router sending most reworded questions to the wrong place, a tool-picker failing on more complex requests, and a date parser that silently breaks on normal language.
More than anything, it changed the conversation from “the system probably works fine” to having actual numbers and evidence to back up, or challenge, that assumption.
