Test AI models v2.0 is hereSee what's new
Test AI models
Blog / Guide

How alerts watch a test, and what they cost

An alert re-runs a test on a schedule and tells you when a cheaper model, a price rise, a slower model, or a changed answer shows up. How to set one up.

Marko MilojkovicFounder, Test AI models
5 min read

Key takeaways

  • An alert watches the winner of a test, and each check re-runs every model the test had and compares the result with that baseline.
  • It reports a cheaper model among the ones you tested, a price rise, a slower model, or a changed answer.
  • Each check uses credits like re-running those models, and only when a check runs.
  • A spend cap per check, a smaller test, or a weekly schedule keeps the cost down.

An alert re-runs a test you have already run, on a daily, weekly, or monthly schedule, and compares each check with the test's winner. It tells you when a cheaper model, a price rise, a slower model, or a changed answer shows up. Each check uses credits like re-running those models, and only when it runs.

The four steps of an alert

StepWhat happensWhere you see it
1. Pick a winnerThe winner becomes the baselineThe test page
2. Set the alertChoose daily, weekly, or monthly, and a spend capThe New alert page
3. Each check runsEvery model the test had runs again, the winner firstThe alert's history
4. A finding shows upThe alert reports what changedThe alert's page, the email, and the bell

An alert starts from a winner

An alert starts from a test you have already run and the winner you picked. The winner is the baseline, which is why the Set alert button asks you to pick a winner first. You choose how often it checks: daily, weekly, or monthly.

Each scheduled check re-runs every model the test had, the winner first, and compares the new results with the baseline.

It reports four kinds of change

  • A cheaper model: a model in the check now costs more than 10% less than the winner did when you set the alert up. Costs are compared at each model's normal rates, so a time-of-day price cannot fake a change.
  • A price rise: the winner now costs more than 10% more than it did.
  • A slower model: the winner now takes more than one and a half times as long, and at least a second longer.
  • A changed answer: the winner's answer shares less than 40% of its words with its original answer.

A cost change is reported at once. A slower model or a changed answer has to show up on two checks in a row, because a single run can be noisy. These thresholds are today's settings (2026-11-12) and can change.

The catch: an alert tells you something is different, not that it is worse. Whether an answer still meets your bar is what the quality check is for.

Every alert has its own page

The alert's page shows the findings in words, the baseline now against when you set it up, the history of checks, Check now, Edit, and a way to make a different model the new baseline when you switch. Alert emails link straight to that page. The bell tells you when a check finds something, fails, or is skipped.

The page also shows models added to the platform since the last check. New models are only suggested: nothing is added to your test on its own.

An alert can watch a whole workflow

On PRO, an alert can watch a workflow instead of a single test. A scheduled check is then an ordinary workflow run, and you only get an email when something new turns up. A workflow can have one alert.

A check costs about what re-running the test costs

A check re-runs every model in the test, so it costs about what running those models again costs today. Like any run, it needs your balance to cover its "Needed to start" figure to begin. Your balance moves once, when the check ends, by what the models really used. Nothing is charged between checks.

To estimate a month, take the credits one check needs and multiply by the number of checks:

daily alert:   credits per check x about 30
weekly alert:  credits per check x about 4
monthly alert: credits per check x 1

Every check's charge appears in your Credit history on the Analytics page.

Four ways to keep the cost under control

  • Set a spend cap. Each alert can have its own cap on credits per check. A check whose "Needed to start" is over the cap is skipped and recorded, not run.
  • Watch a smaller test. An alert re-runs every model the test had, so a test with fewer models costs less per check.
  • Check less often. Weekly or monthly is enough for a price you only want to keep an eye on.
  • Keep an eye on your balance. When your balance is about to run low, you get an email before each check, for as long as it stays that way.

When not to rely on a check running

Your account runs one test at a time. A scheduled check that finds you in the middle of another run waits up to 45 seconds and is then moved to the next daily pass, with nothing charged. A missed or skipped check is moved to its next due time, never retried straight away. If an exact check time matters, press Check now on the alert's page.

Run your own test

Run a test on your real prompt, pick a winner, and set an alert on it. Setting it up uses no credits.

Find the cheaper model for your own prompt
  • 10k free credits
  • No card
Run your own test

Marko

Questions

Does an alert judge whether answers got worse?

No. It tells you an answer changed. Run the quality check to see whether it still meets your rule.

Why does an alert need a winner?

The winner is the baseline every check is compared with, so the Set alert button asks you to pick one first.

Will an alert switch models for me?

No. It reports what it found. You decide, and you can make a different model the baseline on the alert's page.

How many alerts can you have?

Pay as you go has 1 active alert on a test. PRO has up to 25, on tests and workflows. A paused alert does not count.

Marko MilojkovicFounder, Test AI models

Runs Eterna Creative and built Test AI models after repeated multi-provider LLM integration work for client products.

About the team