Evaluate a memory loop
Build a production-shaped acceptance set for writes, recall, and downstream answers.
1. Sample representative traces
Select redacted conversations and source documents that cover stable preferences, corrections, temporal changes, ambiguous queries, missing facts, and tenant boundaries. Keep development and held-out release sets separate.
2. Grade four layers
| Layer | Question |
|---|---|
| Write | Did the correct durable records commit, with no raw fallback? |
| State | Did update, invalidation, history, and scope behave correctly? |
| Recall | Were the necessary records selected with acceptable noise? |
| Answer | Did the downstream agent complete the task from allowed context? |
3. Freeze configuration
Record code commit, model/provider versions, prompts, filters, storage adapter, top-k, score floors, judge rubric, and retry policy. Save item-level outputs and search traces so a changed aggregate score can be diagnosed.
4. Add operational gates
Measure provider calls, input/output tokens, p50/tail latency, failure rate, event completion, and index visibility. A higher recall score can still be a bad release if it requires impractical context or fails unpredictably.
Use FishMem's public benchmark harnesses as examples—not as a substitute for your acceptance set.
Current public evidence
The frozen 2026-08-24 portfolio compares FishMem with mem0 OSS 3.1.2 on
paired items under the same disclosed configuration.
| Suite | Variant | Paired items | FishMem | mem0 | Delta (95% CI) |
|---|---|---|---|---|---|
| LongMemEval | oracle | 500 | 88.2% | 83.8% | +4.4 pt (+1.0 to +7.8) |
| BEAM | 100k | 400 | 46.7% | 41.0% | +5.7 pt (+1.8 to +9.5) |
| LoCoMo | categories 1–5 | 1,986 | 67.5% | 71.1% | -3.7 pt (-5.8 to -1.5) |
All three rows are part of the release decision. The LoCoMo loss is not
filtered out, LongMemEval oracle is not LongMemEval-S, and the current
artifacts do not support public cost or end-to-end wall-clock comparisons.
FishMem also used materially more retrieved context than mem0 on the two
winning suites, so context efficiency remains an explicit follow-up rather
than a marketing claim.