Case study
November 20, 2026

Cloaked now tests new voice AI models in under 1-hour, not days

Cloaked’s Call Guard uses a voice AI model to screen calls for its customers. Eliza built a call test suite and workflow that cut model testing from days to under an hour, and helped the team find a model and prompt setup that cut missed legitimate calls by 64%.

Eliza
Powered by
Open AI
What Eliza delivered
A call test suite for Call Guard (call scenarios, synthetic test audio, scoring) and a workflow for testing model and prompt changes
Client
Cloaked
Industry
Technology
Function
Product and Engineering
Users enabled
Engineering and product teams
Built on
Cloaked’s evaluation harness (built by Cloaked), OpenAI Realtime models
64%
Fewer missed legitimate calls
With a new evidence-first prompt setup
<1 hr
To validate a model change
Down from days. The full stress test takes about 4.5 hours
11%
Lower cost per call
From the new evidence-first prompt, which also answered callers 21% faster
“When you screen more than a million calls a day, you can’t guess whether a new model is better. Eliza helped us get ahead of the unknown unknowns and gave us the right foundation to move fast. Now we test every new OpenAI model in under an hour, before a single member hears it.”
Kyler Ross
Head of Product @Cloaked
The problem

Change the model with confidence and catch issues before rollout

Call Guard answers calls for Cloaked’s customers and decides what to do with each one. The aim is for every real call to reach the customer and no spam or scam call to get through. It sounds simple, but it’s one of the hardest problems in call screening. At more than a million calls a day, even a small regression reaches a lot of people. It runs on OpenAI’s Realtime model, an AI voice model that talks with the caller live.

When the model changes, call handling can change with it, and a regression means unhappy customers and churn. Validating a change took days, and there was no repeatable way to see which calls a new model or prompt (its written instructions) would block or let through before it shipped.

The solution

A call test suite and workflow for every model and prompt change

Over six weeks, Eliza worked inside Cloaked’s engineering and product team to build a call test suite for Call Guard and a workflow for using it. Eliza defined the call scenarios and failure labels, built a synthetic audio pipeline to generate realistic test calls, and designed the experiments and scoring. That let the team test any model and prompt combination on the same calls in under an hour instead of days, before it reached customers. Faster testing is how they found a setup that misses fewer legitimate calls and costs less per call. The test suite runs on Cloaked’s own evaluation harness, the system that runs the tests, which Cloaked built and owns along with its secure data infrastructure. All test calls are synthetic, so no customer recordings were used.

How it works

Every model and prompt change proven before customers hear it

Every model, prompt, or routing change goes through the same five steps before it reaches customers.

Define the synthetic test calls: Hundreds of call scenarios, from a wanted call to a spam robocall, each voiced in conditions like a busy airport or a patchy line. Run them through Cloaked's evaluation harness: Replays Call Guard's call screening outside production and sends the same calls through the current and candidate models. Score every call: AI judges check whether each call was rightly screened as spam or passed to the customer. People review their scores so the team can trust them. Compare models side by side: See how each model and prompt handled the same calls, with quality, speed, and cost per call next to each other and every failed call traced. Decide whether to ship, tune, or hold: The team refines the model, prompt, or routing with evidence. Rollout stays under Cloaked's control.

Re-run and regrade after every change, on the same calls, so results stay comparable.

In practice

How the team uses it today

Cloaked now runs its own experiments with the system. Instead of waiting days for manual checks, the team can see which calls a change would block or let through before anything ships.

  • Testing a new model release. Run the current Realtime model and newer releases on identical calls. Every newer Realtime release tested delivered more legitimate calls than Cloaked’s current setup.
  • Comparing prompt and model combinations. Test prompt versions across models together. A new evidence-first prompt made screening more accurate, answered callers 21% faster, and cut cost per call by 11%.
  • Checking hard call conditions. Generate test calls from a bus, a busy airport, or a weak connection to see where screening breaks down.

Since September, the Call Guard team has kept building the harness inside Call Guard’s main codebase, adding multi-turn caller simulation, label-blind judges, resumable runs, and a tool to compare runs. They use it to test newer OpenAI voice models as they come out, and it showed exactly what Call Guard needs to change to support them, without any guess-and-check in production. Next, the team will add one more check before any new model takes live calls: run a sample of recent Call Guard screenings through the harness and review every call the new model would handle differently.

The results

From days of testing to answers in under an hour

With the test suite and workflow Eliza built, checking a model change now takes under an hour instead of days. In six weeks, the team compared 11 OpenAI model and prompt setups side by side on the same calls. Faster testing paid off: the best setup, “Realtime 2.1 High” with a new evidence-first prompt, delivered 96.5% of legitimate calls, up from 90.2%, so 64% fewer were missed. The new prompt also answered callers 21% faster and cut cost per call by 11%.

Next, the team is tuning toward 100% legitimate-call delivery and adding production monitoring.

Outcome Before: Current Setup After: Realtime 2.1 High + New Prompt
Legitimate calls delivered 90.2% 96.5% 64% fewer missed
Setups compared before a decision No repeatable test 11 side by side on the same calls
Time to validate a change Days Under an hour Full stress test: about 4.5 hours
Seeing why a call failed Overall scores only Complete information Every failure traced and grouped by call condition

Results from Cloaked’s evaluation set of realistic calls.

Know which model to trust before you ship it

Eliza helps product and engineering teams build evaluations that turn every model release into a fast, evidence-backed decision.

Contact us
→