MemAware Benchmark Tests AI Memory Beyond Keyword Search

MemAware is an open-source benchmark designed to test whether AI assistants with memory can surface relevant context from past conversations when current queries don't explicitly hint at that information.
How the Benchmark Works
The benchmark contains 900 questions across three difficulty levels. It tests scenarios where relevant context exists in memory but the current question doesn't contain keywords that would trigger a search match. For example: you told your AI assistant about your 45-minute commute months ago, then later ask "What time should I set my alarm for my 8:30 AM meeting?" The assistant should factor in your commute, but searching "alarm 8:30 meeting" won't find conversations about commuting.
Key Findings
- Search barely helps: BM25 search scored 2.8% vs 0.8% with no memory — a tiny improvement that costs 5x the tokens.
- Vector search fails on hard questions: It helps when keywords overlap (6%) but drops to 0.7% on cross-domain connections — the same as no memory. Example hard question: "How should I bid at the charity auction?" should recall a past $800 handbag purchase as a spending baseline, but embedding similarity can't connect these concepts.
- Searching when you shouldn't is expensive: The "always search" pattern reads ~4.7K tokens of results per question regardless of whether they help. Most of the time, the results are irrelevant noise.
The Core Problem
Current AI memory implementations are essentially just search systems. True memory awareness — knowing what information is stored and proactively surfacing relevant context — is a different problem that search alone can't solve.
The benchmark is available for testing different approaches at: https://github.com/kevin-hs-sohn/memaware
📖 Read the full source: r/ClaudeAI
👀 See Also

OpenClaw Client Adds Live API Cost Tracking, Spending Caps, and Granular Agent Controls
OpenClaw Client now features live usage UI with circular progress bars, per-agent spending caps, sub-agent management, skill toggling, and model switching from different providers.

AgentMind: A Claude Code Plugin That Learns and Applies Your Coding Preferences
AgentMind is a Claude Code plugin that observes your coding patterns, learns preferences like tool choices and style rules, and automatically injects that context into future sessions. It uses a six-step core loop and confidence scoring to determine when to apply learned preferences.

civStation: A VLM System for Playing Civilization VI via Natural Language Commands
civStation is a computer-use VLM harness that plays Civilization VI by translating high-level natural language commands into in-game actions. The system uses a 3-layer architecture separating strategy and execution, with support for human-in-the-loop intervention.

Root Cause of Claude Code VS Code Extension Session Title Corruption Identified
A developer has identified the architectural root cause for session title corruption in Claude Code's VS Code extension, affecting 20+ GitHub issues. The problem stems from the extension reading titles via a raw string search in session files, leading to three failure modes.