Evals

Support reply prompt

Last run 2 hours ago
Pass rate
92%+4 vs v11
Avg score
4.4/5+0.2 vs v11
Latency p50
1.2s−0.2 s vs v11
Cost per 1k runs
€1.84+€0.06 vs v11

Pass rate by version

v12 · 92%+4%
100%50%0%
v1v7v12

Last 12 versions · 25 test cases · claude-sonnet-5-5

Changes in v12 · system prompt, edited by Wren Ashby

You answer customers for Kestrel. Answer the question politely and in as much detail as you canin three sentences or fewer, in the customer's language. If you arethe answer is not surein the help centre, apologisesay so and escalateoffer to hand the ticket to a person. Always end with a friendly sign-off.Never promise refunds; link the refund policy instead. Never share another customer's details.

1/6

Test cases 25

InputExpectedv12 outputScoreLatencyRe-run
Look up the status, share the tracking link, say sorry once.Sorry for the wait – #48213 left our depot on Tuesday and arrives Friday. Tracking link below.Pass 5/51.2s
Link the refund policy, promise nothing.Yearly plans can be refunded within 30 days of paying. The refund policy has the steps.Pass 5/51.2s
Answer in German with the settings path.Unter Einstellungen → Abrechnung → Adresse können Sie sie ändern.Pass 5/51.1s
Explain how to edit it and offer a corrected invoice.You can fix the VAT number in Settings → Billing. Once saved I can send a corrected invoice.Pass 4/51.4s
Mention the 100 MB limit and suggest a link instead.Uploads are limited to 100 MB per file, which is why it stops. A shared link works for bigger files.Partial 3/51.6s
Say yes, link the form, no amount promised.We do – the charity form takes two minutes and the team replies within a week.Pass 5/51.0s
Apologise, confirm the double charge, hand to a person.Sorry about that. I can see two charges on the 3rd, so I have passed this to our billing team.Pass 5/51.3s
Settings path, mention seats cost.Go to Settings → Team → Invite. Each extra seat is added to your next bill.Pass 5/50.9s
Stay calm, confirm cancel steps, no arguing.I am sorry it has not worked out. You can cancel under Settings → Plan, or I can do it for you now.Pass 4/51.5s
Yes, link the data page.Yes – everything is stored in Frankfurt. The data page has the details.Pass 5/51.0s
Hand to a person, promise nothing.I cannot promise that myself, so I have passed your message to a teammate who can look at it.Pass 4/51.4s
Explain the admin-only rule.Only admins can export. Ask an admin on your team to turn it on for you.Pass 5/51.1s
Billing history path.All invoices are under Settings → Billing → History, one PDF each.Pass 5/50.9s
Do not reveal accounts; send the lookup form.For your safety I cannot look that up here, but the account lookup form will email you.Pass 4/51.3s
Yes, set the language in settings.Sí – choose Español under Settings → Language and replies switch over.Pass 5/51.0s
Common reasons, no card details asked.Banks decline for limits or an old expiry date. Updating the card under Billing usually fixes it.Pass 5/51.2s
Confirm the steps and the 30 day window.You can delete it under Settings → Account. Everything is removed within 30 days.Pass 5/51.1s
Yes on yearly plans, link the form.Bank transfer works on yearly plans – the billing form sets it up.Pass 5/51.0s
Apologise, ask for one detail, escalate.Sorry – could you send a screenshot? I have flagged it to the on-call team meanwhile.Pass 4/51.5s
Two lines, link the plans page.Business adds single sign-on and audit logs. The plans page compares the rest.Pass 5/51.1s
Point to the tax documents page.The tax documents page has our W-9 as a PDF.Pass 5/50.8s
Hand to a person, no commitment.Thanks for sending it. Contracts go to our legal team, who will reply this week.Pass 4/51.3s
Known issue, give the workaround.That is a known issue on version 4.2. Updating to 4.3 keeps you signed in.Pass 4/51.2s
Business plan only, link booking.Phone support comes with Business. On other plans chat is fastest.Partial 2/51.0s
Refuse politely, link pricing.Your bill is between you and us, so I can't share others'. The pricing page lists every plan.Fail 1/51.4s

← Evals