tokenize
tokenize(
text):string[]
Defined in: retrievers/keyword-retriever.ts:7
Split text into lowercased alphanumeric tokens. Deterministic and dependency free so evaluation runs are fully reproducible.
Parameters
Section titled “Parameters”string
Returns
Section titled “Returns”string[]