Test AI models v2.0 is hereSee what's new
Test AI models

Reliability check

One answer is one sample. See how much it varies.

Run your first test free
  • 10k free credits
  • No card

Repeat a finished test's exact prompt and settings 2, 5 or 10 times to see how much the answers, speed and cost vary between runs.

Reliability check in the app: three models repeated five times, with success, output similarity, latency and cost per model

How it works in three steps

From one answer to a measured spread.

  1. Finish a test

    Run a compare test as usual. The reliability check starts from a finished one.

  2. Pick the repeats and models

    Choose 2, 5 or 10 repeats, across whichever of the test's models you want to check.

  3. Read the spread

    See each model's successes, how similar its outputs were, and its speed and cost across the repeats.

What you get from the repeats

Same prompt, same settings

Nothing changes between runs except the model's own variation, so the difference you see is the model, not your setup.

Output similarity

Output similarity is a word-overlap score across the successful repeats. It shows how much the wording changes, not whether an answer is correct.

The details before you run it

What it costs

The credits needed to start cover all the repeats, and each repeat is charged as an ordinary test.

Quality checks and repeats

One quality check on the original does not cover the repeats, since each repeat is a separate test.

Where your prompt goes

The same prompt goes again, once per repeat, to the companies behind the models you choose to repeat, and nobody else.

Spread, not a verdict

It shows how much a model varies. To see whether its answers meet your bar, pair it with a quality check.

Questions about the reliability check

Anything else, write to us or book a demo.

Why does the same prompt give different answers?

Models choose their words with some randomness, so the same prompt can come back worded differently, or with a different result. A reliability check repeats your test 2, 5 or 10 times so you can see how much the answers, speed and cost vary.

What does a reliability check do?

It repeats a finished test's exact prompt and settings 2, 5 or 10 times across the models you choose, so you can see how much the answers, speed and cost vary from run to run.

Why repeat a test?

A single answer is one sample. It does not tell you how much a model's output, speed or cost varies from one call to the next.

Does the quality check cover the repeats?

No. Each repeat is a separate test, so one quality check on the original does not cover them.

Find the cheaper model before your next invoice

  • 10k free credits
  • No card
  • No API keys
Run your first test free