safety security
Every Memory Backend Mishandled Permission
An on-device assistant benchmark found systems either leaked too much or disclosed too little.
Summary
An on-device assistant benchmark found systems either leaked too much or disclosed too little.
MemArena simulates 50 agents over 15 days and tests five open-weight readers with several memory backends. The authors report that backend choice affected accuracy more than reader scaling, while permission-aware access failed universally: oracle retrieval leaked heavily and other systems were overly reluctant. Results come from a synthetic single-world benchmark, not real private conversations.
Why it matters
An on-device assistant benchmark found systems either leaked too much or disclosed too little.
Limits and context
- Results come from a synthetic single-world benchmark, not real private conversations.
Key claims
An on-device assistant benchmark found systems either leaked too much or disclosed too little.
Qualification: Results come from a synthetic single-world benchmark, not real private conversations.
Evidence: source-2026-08-05-017
Sources
- arXiv preprint 2608.02609arXiv · primary research
Corrections
No corrections have been recorded for this story.