Reliability check
One answer is one sample. See how much it varies.
- 10k free credits
- No card
Repeat a finished test's exact prompt and settings 2, 5 or 10 times to see how much the answers, speed and cost vary between runs.

How it works in three steps
From one answer to a measured spread.
Finish a test
Run a compare test as usual. The reliability check starts from a finished one.
Pick the repeats and models
Choose 2, 5 or 10 repeats, across whichever of the test's models you want to check.
Read the spread
See each model's successes, how similar its outputs were, and its speed and cost across the repeats.
What you get from the repeats
Same prompt, same settings
Nothing changes between runs except the model's own variation, so the difference you see is the model, not your setup.
Output similarity
Output similarity is a word-overlap score across the successful repeats. It shows how much the wording changes, not whether an answer is correct.
The details before you run it
The same companies as the original test, and nobody else.
What it costs
The credits needed to start cover all the repeats, and each repeat is charged as an ordinary test.
Quality checks and repeats
One quality check on the original does not cover the repeats, since each repeat is a separate test.
Where your prompt goes
The same prompt goes again, once per repeat, to the companies behind the models you choose to repeat, and nobody else.
Spread, not a verdict
It shows how much a model varies. To see whether its answers meet your bar, pair it with a quality check.
Questions about the reliability check
Anything else, write to us or book a demo.
Why does the same prompt give different answers?
Models choose their words with some randomness, so the same prompt can come back worded differently, or with a different result. A reliability check repeats your test 2, 5 or 10 times so you can see how much the answers, speed and cost vary.
What does a reliability check do?
It repeats a finished test's exact prompt and settings 2, 5 or 10 times across the models you choose, so you can see how much the answers, speed and cost vary from run to run.
Why repeat a test?
A single answer is one sample. It does not tell you how much a model's output, speed or cost varies from one call to the next.
Does the quality check cover the repeats?
No. Each repeat is a separate test, so one quality check on the original does not cover them.
Find the cheaper model before your next invoice
- 10k free credits
- No card
- No API keys