The recent generative AI boom and advent of large language models (LLMs) has led many to wonder about the evolution of search engines. Will dialogue-based LLMs replace traditional search engines, or will the tendency of these models to hallucinate make them an untrustworthy source of information? Currently, the answer to these questions is unclear, but the quick adoption of AI-centric search systems such as you.com and perplexity.ai indicates a widespread interest in augmenting search engines with modern advancements in language models. Ironically, however, we have been heavily utilizing language models within search engines for years! The proposal of BERT [1] led to a step-function improvement in our ability to assess semantic textual similarity, causing these language models to be adopted by a variety of popular search engines (including Google!). Within this overview, we will analyze the components of such AI-powered search systems.
Search engines are one of the longest-standing and most widely-used applications of machine learning and AI. Most search engines are comprised of two basic components at their core (depicted above):
- Retrieval: from the set of all possible documents, identify a much smaller set of candidate documents that might be relevant to the user’s query.
- Ranking: use more fine-grained analysis to order the set of candidate documents such that the most relevant documents are shown first.
Depending upon our use case, the total number of documents over which we are searching could be very large (e.g., all products on Amazon or all web pages on Google). As such, the retrieval component of search must be efficient — it quickly identifies a small subset of documents that are relevant to the user’s query. Once we have identified a smaller set of candidate documents, we can use more complex techniques — such as larger neural networks or more data — to optimally order the…