business enterprise
Simulated Customers Screened an AI Agent Before Launch
Nubank reports a 36.69-point tNPS gain in one live test and an 8.82-point self-service gain in another.
Summary
Nubank reports a 36.69-point tNPS gain in one live test and an 8.82-point self-service gain in another.
A team used synthetic customers and simulated tool responses to rehearse customer-service agents without touching production backends. Across four deployed versions of Nubank's card-support agents, version-level simulator scores tracked production evaluator scores. Simulation-guided iteration preceded a live A/B test that raised transactional net promoter score by 36.69 points; a separate screen of more than 16,000 simulated conversations selected an open-weight configuration that increased self-service by 8.82 percentage points without a statistically significant tNPS change. These are company-specific results reported by the authors, not proof that simulated testing predicts every production environment.
Why it matters
Nubank reports a 36.69-point tNPS gain in one live test and an 8.82-point self-service gain in another.
Limits and context
- These are company-specific results reported by the authors, not proof that simulated testing predicts every production environment.
Key claims
Nubank reports a 36.69-point tNPS gain in one live test and an 8.82-point self-service gain in another.
Qualification: These are company-specific results reported by the authors, not proof that simulated testing predicts every production environment.
Evidence: source-2026-09-25-002
Sources
- arXiv preprint 2609.30137arXiv · primary research
Corrections
No corrections have been recorded for this story.