tokenizeQuery

tokenizeQuery(query): string[]

Defined in: packages/memory-ltm/src/lexical.ts:81

Split a natural-language query into distinct, lowercased match terms.

Splitting on non-alphanumerics keeps version-ish and identifier-ish tokens usable (pnpm, 11, postgres) while discarding punctuation. Order is preserved and duplicates removed, so the returned length is the denominator lexicalRelevance scores against.

If every token is a stop word the stop list is ignored rather than returning nothing: a query like “how do we do this” should still attempt a match rather than silently retrieve nothing.

Parameters

query

string | null | undefined

Returns

string[]