Reindex embeddings

Reindexing rebuilds the derived vector index from Postgres, the source of truth. Run it after an embedding model change, a vector store loss, or a bulk import. Design details (cursor resumability, per-item failure isolation): Reindex & backfill.

There are two front doors to the same engine: a CLI for operators on the server host, and admin MCP tools for driving it through an MCP client.

Option A — CLI (on the server host)

pnpm --filter mcp-server reindex

The CLI bootstraps the app context directly, so it needs the server’s .env (DATABASE_URL, DEPLOYMENT_PROFILE) but not MCP_ADMIN_TOKEN — shell access to the host is the credential.

Flags:

Flag	Effect
--user <id>	Reindex one user only (default: all users)
--batch-size <n>	Memories per page, 1–1000 (default 100)
--regenerate	Call the embeddings provider per memory instead of reusing stored vectors — required after a model/dimension change
--recreate	Drop and rebuild the entire index first (clean, orphan-free). Destructive and non-atomic — recall is empty until the rebuild completes. Unscoped runs only: ignored with --user, --cursor, or --max
--max <n>	Stop after n memories
--cursor <id>	Resume from a previous run’s reported cursor

Exit code is 0 on a clean pass and 1 when any item failed; the summary prints processed / indexed / skipped / failed.

Option B — admin MCP tools

All five tools are auth: 'admin': the input carries an adminToken that must equal MCP_ADMIN_TOKEN (constant-time compared). Both profiles are Postgres-backed, so all five are available in lite and standard alike — including the queued tools, whose job rows live in Postgres — see deployment profiles.

Synchronous — blocks until done, returns the summary. Fine for scoped or small corpora:

{
  "name": "reindex_memories",
  "arguments": {
    "adminToken": "<MCP_ADMIN_TOKEN>",
    "userId": "qp",
    "batchSize": 100
  }
}

Queued — returns a jobId immediately; the job persists progress (and its resume cursor) in Postgres and survives observation:

{ "name": "queue_reindex_memories", "arguments": { "adminToken": "<MCP_ADMIN_TOKEN>" } }

Then poll and manage:

• get_reindex_status — { adminToken, jobId } → state (queued/running/completed/failed), progress counts, cursor.
• cancel_reindex_job — stops the job, preserving its cursor.
• retry_reindex_job — resumes a failed/cancelled job from its persisted cursor, not from scratch.

The queue processes one job at a time by design; queueing a second reindex serializes behind the first. Full parameter tables: MCP tools reference.

Recreate — both reindex_memories and queue_reindex_memories accept recreate: true (admin-gated like the rest of the input): the same drop-and-rebuild the CLI exposes as --recreate, previously CLI-only. It is only honored by an unscoped full reindex — no userId, cursor, or maxMemories — and recall is empty until the rebuild completes:

{
  "name": "reindex_memories",
  "arguments": {
    "adminToken": "<MCP_ADMIN_TOKEN>",
    "recreate": true,
    "reuseExistingEmbeddings": false
  }
}

Choosing reuse vs regenerate

Situation	Mode
Vector store lost/corrupted	reuse (default) — vectors are valid, no provider cost
Embedding model or dimension change	recreate + regenerate (see below)
Imported with EMBEDDING_PROVIDER=local, upgrading to real vectors	regenerate
A few individually stale memories	reembed_memory per id instead of a full pass

Regenerated embeddings are written back to Postgres (the embedding float array) before the vector upsert, so a later default (reuse) reindex carries the new vectors forward instead of silently reverting to the old ones.

Model or dimension change

Changing the embedding model or provider (e.g. the 768-dim nomic-embed-text default → the 1536-dim text-embedding-3-small) changes the index shape. The canonical procedure:

1. Set the new EMBEDDING_PROVIDER / EMBEDDING_MODEL (and VECTOR_DIMENSIONS, if you pin it) and restart the server.

2. Run an unscoped reindex with recreate + regenerate:

   pnpm --filter mcp-server reindex -- --recreate --regenerate

   or reindex_memories / queue_reindex_memories with { "recreate": true, "reuseExistingEmbeddings": false }.

This drops the old-dimension index, regenerates every embedding at the new dimensionality, and persists the new vectors back to Postgres. Recall is empty during the rebuild. Full model/provider details: Configure embeddings.

Verify

• Summary shows failed: 0 (a partial pass logs which items failed — they are counted and skipped, never allowed to corrupt Postgres).
• recall returns semantic hits for known memories.
• Re-running is always safe: upserts by stable id are idempotent.