Test AI models v2.0 is hereSee what's new
Test AI models
Blog / Guide

How to write a pass rule for the quality check

The quality check marks each answer against a rule you write. What a good pass rule looks like, with weak and strong examples you can copy and adapt today.

Marko MilojkovicFounder, Test AI models
4 min read

Key takeaways

  • A pass rule is a few plain sentences that an answer either meets or does not.
  • A good rule names the exact values, fixes the format, and says what counts as a fail, instead of words like "good" or "professional".
  • Write the rule before you read the answers, so it cannot drift toward the answer you already like.
  • An uncertain verdict is never counted as a pass.

A pass rule is a short paragraph that tells the quality check what a correct answer looks like. A good one names the exact values, fixes the format, and says what fails, so a judge model can check it line by line. Write it before you read the answers, and keep it short enough to fit in one paragraph.

A good ruleExample
Says what a correct answer containsThe exact category, total, or date, when there is one right answer
Fixes the formatJSON only, one line per item, five bullets, a word limit
Says what failsExtra text, a missing field, an invented fact
Fits in a short paragraphIf it needs a page, split the job into two tests

The quality check judges every answer against your rule

After a test has run, the quality check asks judge models whether each answer meets your rule. A judge is never from the same company as the model that wrote the answer.

Two judges look at each answer, and a third is asked when they disagree. The verdict is the majority, pass or fail, or uncertain when there is no majority.

The check uses credits, like a test, and you see the credits it needs to start before it runs. Your rule is saved with the test, and the results show which rule they were checked against.

Answers marked PASS against the rule, with the cheapest model that passed at the top.

A weak rule leaves the judge to guess

These examples are made up. The weak rules use words like "correctly" or "on-brand", and the judge has to guess what you meant. The stronger ones can be checked line by line.

JobWeak ruleStronger rule
Sorting tickets"Classifies the tickets correctly."Exact categories per ticket, the format, and what fails
Pulling invoice fields"Extracts the invoice data accurately."Exact values, JSON only, numbers as numbers
Rewriting in a brand voice"Sounds on-brand and professional."A word range, banned words, and every fact kept
Summarizing a call"A good summary."An exact count of bullets and the facts that must stay

Copy a stronger rule and adapt it

Each rule below is the stronger version from the table, ready to paste into the rule field and change to fit your job.

Sorting tickets:

Exactly six lines in ticket order and nothing else. The categories must be: 1 billing, 2 bug, 3 question, 4 billing, 5 bug, 6 question. Any extra text, a wrong category, or a missing line is a fail.

Pulling invoice fields:

Valid JSON only, no code fences. invoice_number EXL-20417, total 408, currency EUR, three line items. Money is a number, not a string. A missing field or extra text is a fail.

Rewriting in a brand voice:

50 to 80 words, no exclamation marks, addresses the reader as you, keeps every fact from the original and adds none.

Summarizing a call:

Exactly five bullets and one line starting with Next step:. Keeps the error code, the file size, and the call-back day. Any invented fact is a fail.

Write the rule before you look at the answers

Once you have seen the answers, it is easy to write a rule that the answer you already like happens to pass. Write the rule first, then run the check.

The catch: a rule written after the fact still looks like a rule, and nothing in the check can tell the difference. Only the order you work in protects you from it.

An uncertain verdict means read it yourself

Uncertain means the judges could not agree, or could not give a clear answer. It is never counted as a pass.

Read that answer yourself. If the rule was the problem, tighten it and run the check again.

When not to use a pass rule

A pass rule works when an answer can be checked against something written down. It is a poor fit when there is no single right answer and the only test is taste, such as choosing between two good headlines. For those, read the answers side by side and pick a winner yourself.

Then try cheaper models

Once some answers pass, the Summary tab ranks the ones that passed by cost. Try cheaper models runs your prompt on cheaper models you have not tried and judges them against the same rule. You see the cost first, and each press runs one batch.

Find the cheaper model for your own prompt
  • 10k free credits
  • No card
Run your own test

Marko

Questions

Can you change the rule after the check?

Yes. Edit it and run the check again. The results show which rule they were checked against.

Does the judge see which model wrote the answer?

No. A judge sees your prompt, your rule, and the answer, and it is never from the same company as the model that wrote the answer.

Does a pass mean the answer is always right?

No. It means the judges found that the answer meets your rule. Keep reading a sample of answers yourself, especially for a rule you just wrote.

Marko MilojkovicFounder, Test AI models

Runs Eterna Creative and built Test AI models after repeated multi-provider LLM integration work for client products.

About the team