Cloaked now tests new voice AI models in under 1-hour, not days
Cloaked’s Call Guard uses a voice AI model to screen calls for its customers. Eliza built a call test suite and workflow that cut model testing from days to under an hour, and helped the team find a model and prompt setup that cut missed legitimate calls by 64%.
Change the model with confidence and catch issues before rollout
Call Guard answers calls for Cloaked’s customers and decides what to do with each one. The aim is for every real call to reach the customer and no spam or scam call to get through. It sounds simple, but it’s one of the hardest problems in call screening. At more than a million calls a day, even a small regression reaches a lot of people. It runs on OpenAI’s Realtime model, an AI voice model that talks with the caller live.
When the model changes, call handling can change with it, and a regression means unhappy customers and churn. Validating a change took days, and there was no repeatable way to see which calls a new model or prompt (its written instructions) would block or let through before it shipped.
A call test suite and workflow for every model and prompt change
Over six weeks, Eliza worked inside Cloaked’s engineering and product team to build a call test suite for Call Guard and a workflow for using it. Eliza defined the call scenarios and failure labels, built a synthetic audio pipeline to generate realistic test calls, and designed the experiments and scoring. That let the team test any model and prompt combination on the same calls in under an hour instead of days, before it reached customers. Faster testing is how they found a setup that misses fewer legitimate calls and costs less per call. The test suite runs on Cloaked’s own evaluation harness, the system that runs the tests, which Cloaked built and owns along with its secure data infrastructure. All test calls are synthetic, so no customer recordings were used.
Every model and prompt change proven before customers hear it
Every model, prompt, or routing change goes through the same five steps before it reaches customers.

How the team uses it today
Cloaked now runs its own experiments with the system. Instead of waiting days for manual checks, the team can see which calls a change would block or let through before anything ships.
- Testing a new model release. Run the current Realtime model and newer releases on identical calls. Every newer Realtime release tested delivered more legitimate calls than Cloaked’s current setup.
- Comparing prompt and model combinations. Test prompt versions across models together. A new evidence-first prompt made screening more accurate, answered callers 21% faster, and cut cost per call by 11%.
- Checking hard call conditions. Generate test calls from a bus, a busy airport, or a weak connection to see where screening breaks down.
Since September, the Call Guard team has kept building the harness inside Call Guard’s main codebase, adding multi-turn caller simulation, label-blind judges, resumable runs, and a tool to compare runs. They use it to test newer OpenAI voice models as they come out, and it showed exactly what Call Guard needs to change to support them, without any guess-and-check in production. Next, the team will add one more check before any new model takes live calls: run a sample of recent Call Guard screenings through the harness and review every call the new model would handle differently.
From days of testing to answers in under an hour
With the test suite and workflow Eliza built, checking a model change now takes under an hour instead of days. In six weeks, the team compared 11 OpenAI model and prompt setups side by side on the same calls. Faster testing paid off: the best setup, “Realtime 2.1 High” with a new evidence-first prompt, delivered 96.5% of legitimate calls, up from 90.2%, so 64% fewer were missed. The new prompt also answered callers 21% faster and cut cost per call by 11%.
Next, the team is tuning toward 100% legitimate-call delivery and adding production monitoring.
Know which model to trust before you ship it
Eliza helps product and engineering teams build evaluations that turn every model release into a fast, evidence-backed decision.



%20(1).png)
.jpg)

