Eval suitePro
Score a release candidate of a model-backed feature against the version in production: a grid of test cases fills as the judge works, the pass rate climbs beside last release's, and anything that got worse or better links to the exact case.
oxyn add eval-suiteBilling support replies
v12 · Claude Sonnet 4.5vsv11 · GPT-5 miniJudgeGPT-5
- Cost
- $0.00
- Time
- 0s
- Run the suite to see cases.
The preview stays free for everyone. Pro opens the code of every Pro component, screen, agent-app template, block and kit, plus the Pro CLI.
Rendered from Rust at build time · goes live when visible
When to use
- Run a new version of an LLM feature against its baseline, case by case, and see the regressions and fixes with the cases behind them.
When not to use
- Sit in on a crew of agents mid-run: use Live Session instead.
- One overview for a whole fleet: use Fleet overview instead.
- The fleet overview laid out against time: use Fleet timeline instead.
Installation
Eval suite is part of Oxyn Pro; the preview is free. Pro items install with the oxyn CLI once your licence is signed in; until then Pro members copy the code from this page.
oxyn add eval-suiteUsage
The preview is free. Pro opens the code of every Pro component, screen and template.
Examples
Ready
Idle, 48 cases queued.
Billing support replies
v12 · Claude Sonnet 4.5vsv11 · GPT-5 miniJudgeGPT-5
- Cost
- $0.00
- Time
- 0s
- Run the suite to see cases.
Code Pro
The preview is free. Pro opens the code of every Pro component, screen and template.
Regressions
Finished, regressions only, one open (Oxyn extra).
Billing support replies
v12 · Claude Sonnet 4.5vsv11 · GPT-5 miniJudgeGPT-5
- Cost
- $0.60
- Time
- 1m 31s
- Input
- Customer asks to speak to a person #4: "I paid for a year in March and want my money back now."
- Expected
- Explain the 30-day rule, offer a refund of the unused months and link the billing page.
v11
Annual plans can't be refunded. You can cancel so it doesn't renew.
v12
Annual plans can't be refunded after 30 days. You can cancel so it doesn't renew next year.
- Correct policy2/5was 4
- Answers the question3/5was 4
- Tone4/5was 4
Misses the refund of the unused months the policy allows.
Code Pro
The preview is free. Pro opens the code of every Pro component, screen and template.
Judge fails
The judge stops halfway (Oxyn extra).
Billing support replies
v12 · Claude Sonnet 4.5vsv11 · GPT-5 miniJudgeGPT-5
- Cost
- $0.00
- Time
- 0s
- Run the suite to see cases.
Code Pro
The preview is free. Pro opens the code of every Pro component, screen and template.
API reference
EvalSuite
| Prop | Type | Default | Description |
|---|---|---|---|
nowrequired | ClockMs | — | Host clock. |
done | bool | Default::default() | Open with the run finished. |
fail | bool | Default::default() | The judge fails halfway (error scenario). |
filter | usize | Default::default() | Start filter (0 all, 1 failed, 2 regressions, 3 improved). |
open | Option<usize> | Default::default() | Open this case at the start. |
Read from the Rust signature and doc comments.
Keyboard interactions
| Key | Action |
|---|---|
| EnterorSpace | Start or halt; expand a case; accept it or file it |
| ←or→orHomeorEnd | Change the case list view |
Accessibility
- Every category grid is exposed as one image with a written tally rather than one stop per cell.
- Outcomes use distinct marks (✓, ×, !) and newly broken cases are filled in.
- Case rows toggle their detail and say so; scores and triage choices are read out.
From the spec
Each category row is one image with a full text summary; pass, fail and error have their own glyphs; case rows are buttons with aria-expanded; the filter is a radio group with arrow keys; results and triage are announced. Every action returns the line the polite live region announces. The focus outline shows only for keyboard focus.
Motion & interaction
Cells pop in with their glyph; regressions are the only solid cells; category bars show a ghost tick at the baseline; history bars let you read earlier runs.
Timing
Everything on screen is derived from the host clock now (no timers in the block), so playback, tests and the GPUI view agree.
| Transition | Duration | Easing |
|---|---|---|
| Case scored | 90 ms each (48 cases) | linear |
| Cell in | 180 ms | ease-out |
Reduced motion: shimmer, pulses and slides stop; every state keeps its words and icons.
Responsive behaviour
- Max 760 px; the grid moves under its label below 680 px and the two answers stack below 560 px.
Edge cases (spec)
Up to 760 px; below 680 px the cells go under each label; summary goes to two columns at 620 px and one at 480 px; outputs stack below 560 px; below 480 px categories, triage words and filter counts hide. Layout follows the block's own width (container queries), never the viewport. Checked at 1440, 768 and 390 px in light and dark with no text overlap. Chrome is monochrome grey; colour is semantic only (success, warning, danger, info) plus data series and the official coloured brand and model marks.
Notes for AI
- Import
EvalSuitefromoxyn_dioxusand includeSTATUS_INDICATOR_CSSplus the token stylesheet once. - A GPUI version exists:
BlocksViewinoxyn_gpui. - Props to set first:
now. - It reads a clock (
now) for live timers; pass one clock for the whole page rather than one per component. - Keep its states and timings as they ship; restyle it through the CSS variables rather than by hand.