Keyword search versus search by meaning
The same question, two ways. Keyword search matches words; meaning search compares embeddings.
Keyword search (BM25)
Scores sentences that contain your exact words, weighting rare words more.
No sentence contains any of your words, so keyword search finds nothing.
Search by meaning (embeddings)
Embeds every sentence and your query, then ranks by cosine similarity.
Loading the recorded results
Challenges
0 of 3 complete
- Write your own search that shares no words with any sentence, so keyword search finds nothing, yet search by meaning finds a match with a cosine of 0.40 or more. (not yet)Needs the model download: it asks for your own search.
- Turn on Edit the sentences and add one that uses your search's words in a different sense, so keyword search ranks it first but search by meaning ranks another sentence first. (not yet)Needs the model download: it asks for your own sentence.
- Search for something none of the sentences is about (two words or more), so the best match by meaning scores below 0.15. (not yet)Needs the model download: it asks for your own search.
Try this
- Find a query where keyword search wins and one where embedding search wins.
- Add a sentence that shares many words with a query but means something else. Which method is fooled?
Keyword search and search by meaning in codePython with NumPy
import re
import numpy as np
docs = [
"Putting a little money aside every month adds up over time.",
"Most cats enjoy climbing and sitting in high places.",
"The museum opens at nine and closes at five on weekdays.",
]
STOP = set("a an and are as at be by did do does for from had has have how in is it its of "
"on or that the this to was were what when where which who why will with".split())
def words(text):
return [w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP]
def bm25(query, docs, k1=1.2, b=0.75):
terms = [words(d) for d in docs]
avg_len = np.mean([len(t) for t in terms])
scores = np.zeros(len(docs))
for w in set(words(query)):
n = sum(w in t for t in terms) # how many documents contain the word
idf = np.log(1 + (len(docs) - n + 0.5) / (n + 0.5)) # rare words count more
for i, t in enumerate(terms):
f = t.count(w)
scores[i] += idf * f * (k1 + 1) / (f + k1 * (1 - b + b * len(t) / avg_len))
return scores
# Search by meaning ranks by cosine similarity of embeddings. The demo gets 384
# numbers per text from MiniLM; made-up 3D vectors (money, animals, opening
# hours) stand in for them here.
doc_vecs = np.array([[0.9, 0.1, 0.2], [0.1, 0.95, 0.1], [0.1, 0.1, 0.95]])
queries = {"how can I save cash": [0.95, 0.0, 0.1], "museum hours": [0.1, 0.0, 1.0]}
for query, q in queries.items():
q = np.array(q)
cosine = doc_vecs @ q / (np.linalg.norm(doc_vecs, axis=1) * np.linalg.norm(q))
print(f"{query!r}")
print(" keyword scores:", np.round(bm25(query, docs), 2))
print(" cosine scores: ", np.round(cosine, 2))'how can I save cash'
keyword scores: [0. 0. 0.]
cosine scores: [0.99 0.11 0.21]
'museum hours'
keyword scores: [0. 0. 1.08]
cosine scores: [0.31 0.11 0.99]