Test AI models v2.0 is hereSee what's new
Test AI models

Quality check

Write the rule. Other models judge every answer, LLM as a judge.

Run your first test free
  • 10k free credits
  • No card

Describe what a good answer looks like in plain words. Two cheap judge models mark every answer pass or fail, and a third breaks a tie. Then see the cheapest model that passed.

Quality check in the app: the cheapest model that passed the check, with every checked answer marked PASS

How it works in three steps

From your prompt to the cheapest model that passes your rule.

  1. Run a test

    Paste your prompt and run it on the models you pick. The quality check works on a finished test.

  2. Write your pass rule

    Say what a good answer is in your own words, for example "answers in under 50 words and doesn't mention competitors."

  3. Read the verdicts

    Each answer comes back marked pass, fail or uncertain, and the cheapest model that passed is shown first.

What you get after a check

Cheapest that passes

The answers that passed are ranked by cost, so the cheapest passing model is the first thing you see.

Try cheaper models

Run the same prompt on a few cheaper models you have not tried and judge them with the same rule. The cost is shown first.

Your own rule

You write what counts as a pass. The rule is remembered on the test, so checking again only bills the judging.

Judges from another company

A judge is never from the same provider as the answer it judges.

Uncertain is never a pass

No majority between the judges means the answer is marked uncertain. It is a check, not a guarantee.

The details before you run it

What it costs

Before a check starts, your balance has to cover every judge call that could happen. Afterwards you are charged once, for what the judging cost. Most answers only need the two cheaper judges.

Checking again

Your rule is remembered on the test. Edit it and check again any time: that bills only the judging, never the original answers.

Where your prompt goes

Your prompt, your pass rule and each answer go to two or three judge models from Anthropic, xAI and Google, companies based in the United States. You do not pick these, and a judge never comes from the company that wrote the answer.

What it does not cover yet

A reliability check's repeats are separate tests, so one quality check on the original does not cover them.

Questions about the quality check

Anything else, write to us or book a demo.

What is LLM as a judge?

Using AI models to grade AI answers. In the quality check you write the rule in plain words; two cheap judge models mark each answer pass or fail, and a third breaks a tie. When the judges cannot agree, the answer is marked uncertain, never passed.

Do I need code to evaluate LLM answers?

No. You write the rule in plain words in the app and run the check on a finished test. No code, SDK or API keys.

What does the quality check do?

You write a pass rule in plain words, for example "Replies with only one word: billing, bug or question." Two cheap judge models read each answer and mark it pass or fail. If they disagree, a third decides. If there is still no majority, the answer is marked uncertain, never a pass. A judge never comes from the same company as the answer it judges. It is a check against your rule, not a guarantee, so look closely at the answers that matter.

What happens when the judges disagree?

A third judge is brought in automatically and the majority decides. If there is still no majority, the answer is marked uncertain instead of being forced into a pass or a fail.

Can a judge favor its own company's model?

No. A model is never used to judge an answer from its own provider.

Does checking again re-run my test?

No. Checking again never re-runs the test or re-bills the answers, only the judging.

Is a pass a guarantee?

No. It is a check against your rule, not a guarantee, so look closely at the answers that matter.

Find the cheaper model before your next invoice

  • 10k free credits
  • No card
  • No API keys
Run your first test free