TLDR:

GraphRAG (when fully created and retrieved in Neo4j via Cypher) enhances faithfulness (a RAGAS metric that is similar to precision - e.g. does it accurately reflect the information in the RAG document) over vector-based RAG, but does not effect other RAGAS metrics. It may not offer enough ROI to justify the hype of the accuracy benefits given the performance overhead.

Implications (see list of potential biases in this analysis at bottom of post):

1. Improved accuracy: GraphRAG could be beneficial in domains requiring high precision, such as medical or legal applications.
2. Complex relationships: It may excel in scenarios involving intricate entity relationships, like analyzing social networks or supply chains.
3. Trade-offs: The improved faithfulness comes at the cost of increased complexity in setup and maintenance of the knowledge graph, so the hype may not be justified.

Introduction:

This post is a follow up to GraphRAG Analysis Part 1, which performed RAG on the US Presidential Debate transcript between Biden and Trump (a document not in the training data of any model as of this blog post), comparing the vector db of Neo4j (a graph database) to the vector db of FAISS (a non-graph database). This allowed for a clean compare on the database, while in this post (being Part 2), the comparison incorporates knowledge graph creation and retrieval in Neo4j using cypher against the FAISS baseline to evaluate how these two approaches perform on RAGAS metrics for the same document.

Code runthrough below, notebook hosted here on my Github.

Setting Up the Environment

First, let's set up our environment and import the necessary libraries:

Setting Up Neo4j Connection

To use Neo4j as the graph database, let's set up the connection and create some utility functions:

These functions help us manage our Neo4j database, ensuring we have a clean slate for each run and that our vector index is properly set up.

Data Processing and Graph Creation

Now, let's load our data and create our knowledge graph:

This approach uses GPT-3.5-Turbo to extract entities and relationships from our text, creating a dynamic knowledge graph based on the content of our document.

Setting Up Retrievers

We'll set up two types of retrievers: one using FAISS for vector-based retrieval, and another using Neo4j for graph-based retrieval.

The FAISS retriever uses vector similarity to find relevant information, while the Neo4j retrievers leverage the graph structure to find related entities and their relationships.

Creating RAG Chains

Now, let's create our RAG chains:

These chains associate the retrievers with a language model to generate answers based on the retrieved context.

Evaluation Setup

To evaluate our RAG systems, we'll create a ground truth dataset and use the RAGAS framework:

This setup creates a ground truth dataset, evaluates our RAG chains using RAGAS metrics, and visualizes the results.

Results and Analysis

This analysis revealed a surprising similarity in performance between GraphRAG and vector-based RAG across most metrics, with one difference:

1. Faithfulness:
2. Neo4j GraphRAG significantly outperformed FAISS (0.54 vs 0.18)

The graph-based approach excels in faithfulness likely because it preserves the relational context of information. When retrieving information, it can follow the explicit relationships between entities, ensuring that the retrieved context is more closely aligned with the original structure of the information in the document.

Implications and Use Cases

While the overall performance similarity suggests that for many applications, the choice between graph-based and vector-based RAG may not significantly impact results, there are specific scenarios where GraphRAG's advantage in faithfulness could be crucial:

1. Faithfulness-critical applications: In domains where maintaining exact relationships and context is crucial (e.g., legal or medical fields), GraphRAG could provide significant benefits.
2. Complex relationship queries: For scenarios involving intricate connections between entities (e.g., investigating financial networks or analyzing social relationships), GraphRAG's ability to traverse relationships could be advantageous.
3. Maintenance and updates: Vector-based systems like FAISS may be easier to maintain and update, especially for frequently changing datasets.
4. Computational resources: The similar performance in most metrics suggests that the additional complexity of setting up and maintaining a graph database may not always be justified, depending on the specific use case and available resources.

Note on Potential Biases:

1. Knowledge graph creation: The graph structure is created using GPT-3.5-Turbo, which may introduce its own biases or inconsistencies in how entities and relationships are extracted.
2. Retrieval methods: The FAISS retriever uses vector similarity search, while the Neo4j retriever uses a Cypher query. These fundamentally different approaches may favor certain types of queries or information structures, but this is what is being evaluated.
3. Context window limitations: Both methods use a fixed context window size, which may not capture the full complexity of the knowledge graph structure if anything different is required.
4. Dataset specificity: Overall (and this is a given in 100% of all AI tool analysis): the analysis is performed on a single document (debate transcript), which may not be representative of all potential use cases.

Originally posted at: