Memory for an AI companion

A companion that chats for weeks loses the thread. Re-feeding the entire history into every reply is slow and expensive, and the window fills up long before the conversation is over. The relationship's continuity — the user's name, their dog, what they were stressed about last week — drowns in raw transcript.

Extract the durable facts from casual chat instead, scope them to the user, and recall the few that matter before each reply. Recall is fast enough to fit inside the turn budget: search latency is p50 ~420ms, so it adds little to the time the user waits for a response.

The loop: add on every turn, search before reply

Scope to user_id, use infer: true so FishMem pulls the durable facts out of casual conversation, and keep top_k small so you recall what's relevant without padding the prompt.

import { FishMem } from 'fishmem';
import { llm } from './llm'; // your model call

const client = new FishMem({ apiKey: process.env.FISHMEM_API_KEY });

async function reply(userId: string, message: string) {
  // 1. Search relevant memories before answering.
  const { results } = await client.search(message, {
    user_id: userId,
    top_k: 5,
  });

  const memory = results.map((m) => `- ${m.memory}`).join('\n');

  const system = [
    'You are a warm, consistent companion. Use what you remember about this person.',
    'Recalled context:',
    memory || '(nothing yet)',
  ].join('\n');

  const answer = await llm({ system, user: message });

  // 2. Add the turn so durable facts get extracted for next time.
  await client.add(
    [
      { role: 'user', content: message },
      { role: 'assistant', content: answer },
    ],
    { user_id: userId } // infer: true is the default
  );

  return answer;
}

Over time the casual mentions — "my dog Biscuit had surgery," "I started a new job at a hospital" — become durable facts the companion can bring up naturally, without ever re-reading the full chat log.

Keep recall inside the turn budget

• Latency — search runs at p50 ~420ms, so a single recall comfortably fits inside the time the user already waits for a model response. Run the search and the model call in the order above; the add can happen after you've replied.
• Keep top_k small — a companion needs the handful of facts that bear on the current message, not everything on file. A small top_k keeps the prompt tight and recall sharp. Hybrid recall ranks the results so the most relevant memories come first.