In the vast expanse of today’s data-driven world, finding specific information can be like searching for a needle in a haystack. Imagine that you have a huge dataset, such as Wikipedia pages detailing Marvel movies and TV shows, and you need to find specific information — for instance, details about the parents of Tony Stark (Iron Man). Sifting through countless pages is daunting and inefficient. This is where the power of semantic search comes into play. Semantic search systems, shown in image 1 below, are designed to understand the context and semantics of your query, offering you precise information without the hassle of endless scrolling.

Let’s continue with our Ironman example from the Marvel dataset. Imagine a scenario where a fellow fan is curious about the intricate details of Tony Stark’s character development throughout the Marvel Cinematic Universe. A traditional keyword-based search might return a mix of general Iron Man movie summaries or unrelated character profiles. For example, searching for Ironman character development won’t return results about Tony Stark, but may return results without much context around it. However, with a sophisticated semantic search system, the fan can query something as specific as “Tony Stark’s character evolution in the MCU” and receive targeted, context-rich results. This precision brings the user directly to the most relevant scenes, dialogues, and character arcs, offering a tailored and enriched fan experience.

Now, one may ask: How does it work, technically? Where should we store a large amount of unstructured data so that it can be easily queried and information can be retrieved from it? To answer these questions, I worked on an example project in which I gathered a collection of research papers and abstracts. I then created a semantic search engine capable of addressing my queries about these research papers. My goals were twofold. First, I wanted to understand how the system works behind the scenes and how it is an upgrade from a simple keyword-based search. Second, I wanted to see how different tools like LLMs and vector databases can be used to create a semantic search application.

Here are some technical nitty-gritty details of this project:

1. Dataset: https://paperswithcode.com/dataset/arxiv-10
2. Database: Milvus-lite vector database
3. Text embeddings: Cohere Embed

At the heart of semantic search is data representation. To make unstructured data accessible for machines to read, it needs to be represented in a numerical or vector format. This is where vector databases come in handy. Let’s explore what a vector database is.

Vector Databases

What is a vector database? Simply put, a vector database is a database that stores vector representation of unstructured data and enables search and information retrieval of the data, along with metadata, in an efficient manner. This blog on vector databases provides more technical details. In this project, I have used Milvus-lite, a lightweight version of the Milvus vector database. I chose Milvus for the following reasons:

1. It’s an open-source vector database
2. It’s a powerful tool to use for applications like semantic search, given the dense nature of the dataset that contains millions of vectors
3. Its architecture is distributed, enabling horizontal scalability that separates storage and computing
4. Milvus-lite helps get you started quickly

Here’s how I got started:

Here, I defined the schema of the collection to put in the vector database. I made “id” the primary key, added the title, abstract, and label (describes the category the paper falls into, for example, ‘physics’) columns from the dataset, and set their maximum length. Initially, I had arbitrarily set these values, but when I tried to run the entire code, I got errors. The best way to tackle this would be to calculate the max_length of the columns separately and then initialize it in the schema.

The next step was to initialize the index. To search or query anything in the database, you need an index. In this specific case, the choice was to implement the IVF_FLAT (Inverted File Index) because it offers faster search times, particularly with high-dimensional data like that used in this project. We used the IVF_FLAT index type, especially since I ran this on a CPU. Vector indexes like IVF_FLAT (inverted file index) are employed for approximate nearest neighbor (ANN) search algorithms, which aim to locate the closest points in a high-dimensional space. Here are the top resources that explain ANN in more detail:

1. Activeloop’s article provides a simple definition of how ANN works and has more reading resources to understand the algorithm in detail. https://www.activeloop.ai/resources/glossary/approximate-nearest-neighbors-ann/
2. In this Zilliz’s article, they mentioned when to use ANN along with a straightforward explanation of the algorithm: https://zilliz.com/glossary/anns

To calculate the similarity distance between the target input and the center of each cluster, the distance metric used was L2, which is the Euclidean distance. I set `nlist` to 100 (as seen in the following code snippet) as I had over 8K records of data, so I wanted to divide the data into 100 clusters.

Once the database was ready, the next step was to create the vector embeddings of the dataset. Here, I called Cohere’s client since I used Cohere’s new version of the Embed model. Before I get into the code, let’s discuss Cohere’s embedding model.

Here are the reasons why I chose to use the Cohere Embed model:

1. Enables much faster search capabilities
2. Returns better quality of matched responses
3. Its new compression-aware training method helps reduce the cost of running vector databases

To give a rough idea of pricing, I embedded just the 10,000 abstracts column from the dataset which equaled 22,315,570 tokens and cost me $2.232 (1,000,000 tokens/$1.00).

Here, I used Cohere’s client API because I used the new version of Cohere’s Embed model:

In this version of the Embed model, there is an extra parameter, `input_type`, where you tell the function whether you are embedding a search document or a search query. This additional parameter ensures the highest quality of the user search query and performs more efficiently. Once the embeddings are ready, you must insert them into the collection you created above.

Note that I have just embedded the abstract since the search I intend to do is to ask about a bunch of papers on topics that are included in the dataset. Once the core of the application was set, the next step was to provide the list of search terms, call the `embed` function, and wait for the results.

The setup is straightforward. I wanted to see the title of the paper and abstract that are the most similar to the search term. If you recall, when I declared the variables before creating the collection, I had set the value of `topK = 5` since I wanted to see the top five answers to my search term. The model returns the similarity score, along with the results, and ranks the results accordingly. Image 2 shows a sample result returned from this model:

The overall process of building semantic search applications is simple, especially with all the tutorials out there. It does take some trial and error to understand what is going on in the code fully. Is my code perfect? No. There is a large scope for improvement, for example:

1. Playing with the values of `nlist` and `nprobe`
2. Using a larger version of the Embed model
3. Adding more data
4. Adding a RAG component to the application

If you’re interested in building your semantic search system, I’d strongly recommend that you start simple, and then build upon new features step by step. For the entire code, check out my GitHub repository.