Alerts
LLM monitoring on your own test. Get told when a model changes.
- 10k free credits
- No card
An alert re-runs a test on a schedule and tells you in words when a model's cost, speed or answer changed. It also points out a cheaper model among the ones you tested.
How it works in three steps
From one finished test to a check on a schedule.
Pick a test or a workflow
Put an alert on a finished test with a winner picked, or on a whole saved workflow on PRO.
Choose the schedule
Daily, weekly or monthly, with an optional cap on credits per check.
Read the findings
Findings arrive by email and in the bell, in words, with a link to the alert.
What you get on every check
Cost, speed and answers
Each check re-runs the test and compares it with your original. Cost changes are reported straight away. A slower model or a changed answer is reported after two checks in a row.
Alerts on whole workflows
On PRO, an alert can watch a whole workflow. Each check re-runs every step and compares it with the last one, so you see what changed in cost, speed and answers.
Daily, weekly or monthly
You pick the schedule. Cost changes are reported straight away. A slower model or a changed answer is reported after two checks in a row.
Spend caps
Set a cap per check or per workflow run. If the estimate is over the cap, nothing runs.
Email and in the app
Findings arrive by email and in the bell, in words, with a link to the alert.
The details before you set one
The models you saved on the test, and nobody else.
Alerts per plan
Pay as you go includes 1 active alert, on a test. PRO includes up to 25, on tests and workflows.
What it costs
Each check is a real run, estimated first. A check above its cap is skipped and recorded, not run.
Where your prompt goes
The same prompt goes again, on each check, to the companies behind the models the alert uses, and nobody else.
Change, not quality
An alert tells you something is different, not that it is worse. Judging whether an answer still meets your bar is what the quality check is for.
Questions about alerts
Anything else, write to us or book a demo.
Is this the same as production LLM monitoring?
No. An alert re-runs your saved test on a schedule and compares it with the winner you picked. It does not trace your app's live traffic.
What happens when a model is retired?
If a model in your alert is retired and has a successor, the alert moves to the successor and tells you with a notice.
What does an alert check?
Each check re-runs the test, or the workflow, and compares the new result with your original: cost, speed, and whether the answer looks meaningfully different.
When is a change reported?
Cost changes are reported straight away. Speed and answer changes have to show up on two checks in a row, since a single run can be noisy.
How many alerts can I have?
Pay as you go includes 1 active alert, on a test. PRO includes up to 25, on tests and workflows. A paused alert does not count.
What happens to my alerts if PRO ends?
Nothing is deleted. Alerts over your plan's limit are paused, keep their history and start again if you start PRO again.
Find the cheaper model before your next invoice
- 10k free credits
- No card
- No API keys