06 / AI / Production system
A production AI support agent
A customer-facing RAG agent tuned on real query logs, with an escalation path and a human takeover dashboard. Live since March 2026.
- Status
- No agentchanged toLive since March 2026
- Answer grounding
- Generic model outputchanged toRAG on real query logs
- Escalation
- No path off the botchanged toHuman takeover, RBAC
- Verification
- Manual spot checkschanged toPassing test suite
Challenge
Customer support for this product is mostly the same forty questions, asked in a thousand different ways, about a product people apply to their skin. That combination is exactly why a generic chatbot is the wrong answer. Repetition makes automation worth it, and regulation makes a confident wrong answer expensive.
The system had to be right, it had to know when it was out of its depth, and a human had to be able to take the conversation over without the customer starting again.
What I did
- Built the retrieval layer on the real query logs, so the agent is tuned on the language customers actually use rather than on the language the brand uses.
- Set the escalation rules explicitly. The agent hands off on low confidence, on anything that touches a claim it is not allowed to make, and on any request a human should own.
- Built the human takeover dashboard: real-time conversation view, role-based access control, and a clean handover so the agent stops talking the moment a person picks the conversation up.
- Wrote the test suite, because a support agent that regresses quietly is worse than no agent.
Result
- Live and customer-facing since March 2026.
- Answers grounded in the brand's own documentation and query history.
- A working escalation path rather than a dead end, with the team able to step in mid-conversation.
- Tests passing, so changes to the agent are made with evidence rather than hope.