We test LLM support bots, then we publish what we find.
Six studies run on real models in November 2026: AI disclosure after a human handoff, red-team failures, the LLM bill, model switches, invented policy and the cost of a correct answer. And four case studies built from public records: agent hijacks, documented incident costs, CVEs in AI frameworks and the control gap in real incidents. Every report publishes its method and data; HandoffBench and MigrationBench also ship their code.
Do support bots stay honest after a human handoff?
HandoffBench: does the bot say it is an AI after the platform hands the chat to a named human agent? 11 models from 3 providers, 5 languages, plus a code-level fix.
Never said they were AI after a named handoff, % of replies, 11 models
40 attacks, 5 models: model failure or missing guardrail layer?
The same 40 attacks on a fictional bank bot across 5 models, with and without a guardrail layer.
Case-runs failed, % per model (5): amber without, white with the guardrail layer
The LLM bill: what actually cuts cost
The Cost Cutter demo's synthetic support workload, token for token, under six cost levers, with the eval kept on.
Monthly bill after each of the six levers, US$: amber before, white after
What breaks when you switch models?
One old model, seven replacements, 320 items: grounded support answers, strict-schema tool calls, invoice extraction. Paired item by item.
Items each of the 7 replacements broke (amber) and fixed (white)
How often do AI support bots invent a policy?
Seven models, one fictional store’s written policy, 160 trick questions each, two judges from different providers.
Invented a policy, % of 160 trick questions, by model (7)
What does a correct answer cost?
160 fictional support and invoice items on nine models, list price × measured tokens, plus routing and caching.
Cost per 1,000 correct answers, US$, by model (9)
Case studies from public records
Built from vendor advisories, CVE databases, court and regulator records and the AI Incident Database, with every source linked from the case file.
How real AI assistants and agents were hijacked
59 publicly disclosed hijacks of shipped AI products, each with a primary source: where the instruction entered and what a tool gate would have done.
Cases by where the hijacking instruction entered
What AI incidents actually cost, in documented money
44 documented costs in five categories that are never added together: fines, court sanctions, damages, fraud and stolen-key bills (plus 2 one-day market-value moves, shown for scale).
Median documented cost by category (5), US$, log scale
Published CVEs in popular AI frameworks, agent tools and MCP packages
Every CVE published since January 2023 in 22 widely used AI products, from NVD and the GitHub Advisory Database.
CVEs published 1 Jan–12 Nov of each year, 2023 to 2026
The control gap in real AI incidents at organisations
Well-documented incidents where an organisation’s own chatbot, assistant or agent caused harm: what was in place, and the cheapest control that would have stopped it.
Incidents by the cheapest control that would have stopped them
How we work
In the studies, every number comes from runs we made, on the date stated, with the raw outputs linked from each report. Test companies and people in our benchmarks are fictional, and we do not attack third-party systems. When a run had a problem (a rate-limited model, a judge we had to validate) the report says so in its method section. The case studies use public records only, with inclusion rules written before collection and every quote checked against its source.
KH Labs fixes AI support bots, cuts AI bills and helps teams switch AI models safely, and we publish how we test. Write to us at hello@khlabs.dev.