MS MARCO click-translation expansion tables ("poor man's" DSSM) [P]

TLDR: I made "poor man’s" DSSM (Deep Structured Semantic Model) — the count-based translation table that can enrich the inverted index for full-text search. This trick can improve baseline BM25.

So the idea is the following:

- You have supervised pairs (query, relevant document), e.g., MS MARCO or click logs.

- You tokenize both sides into some units (char n‑grams, wordpieces, words).

- You count cross‑pair co‑occurrences: unit u on the document side vs. unit v on the query side (not co‑occurrence within the same text).

- For each document‑side unit u, you keep the top‑k query‑side units v with the strongest association.

- At indexing time, each document gets postings not only for its own units, but also for the top‑k associated units of each of its units — i.e., document expansion baked into the inverted index.

It’s like mixing synonyms into the search query (but it’s not a synonyms exactly). The one difference from the DSSM is that it can only handle linear dependencies whilst DSSM can do the non-linear one.

And so it improves the performance over BM25 baseline.

I packed it as hf model repo: https://huggingface.co/mirth/msmarco-expansion-tables with a small usage demo script.

I am not claiming that this is a new idea. I made it because it’s fun and I’m planning to use it in my own search engine project.