Graph-Based RAG: Building Knowledge Graphs with Neo4j and LangChain

Standard vector search often hits a wall when your data has deep, multi-hop relationships. If you ask a question that requires joining information across documents—like "How does the recent change in our API schema affect the legacy authentication module?"—a standard semantic search will likely return isolated chunks. It misses the structural connections. This is where Graph-Based RAG (Retrieval-Augmented Generation) changes the game by mapping entities and their relationships into a Neo4j knowledge graph.

Why Graphs Matter for RAG

When I build RAG pipelines, I treat vector search as the "what" and graph search as the "how." Vectors are great for finding similar content, but they lack context about hierarchy, ownership, or causality. By using Neo4j, we create a structured map of our domain. When the LLM queries this graph, it isn't just guessing based on embeddings; it’s traversing a path of verified relationships.

The Architectural Flow

My standard setup for this involves three distinct phases:

1. Extraction: Using an LLM to parse unstructured text into nodes (Entities) and edges (Relationships).
2. Ingestion: Loading these into Neo4j.
3. Retrieval: Combining Cypher queries with vector similarity to provide the LLM with a hybrid context.

Implementation: Extracting and Storing Relationships

I’ve found that using langchain-community’s LLMGraphTransformer is the most efficient way to turn raw text into graph data. Here is a practical snippet of how I structure this pipeline.

from langchain_experimental.graph_transformers import LLMGraphTransformer
from langchain_openai import ChatOpenAI
from langchain_community.graphs import Neo4jGraph
from langchain_core.documents import Document

# Initialize the graph connection
graph = Neo4jGraph(
    url="bolt://localhost:7687",
    username="neo4j",
    password="your_password"
)

# Use a structured output-capable model
llm = ChatOpenAI(temperature=0, model_name="gpt-4o")
llm_transformer = LLMGraphTransformer(llm=llm)

# Sample text from a technical spec
text = "The AuthModule depends on the UserDB. The UserDB handles OAuth2 tokens."
doc = [Document(page_content=text)]

# Extract graph data
graph_documents = llm_transformer.convert_to_graph_documents(doc)

# Save to Neo4j
graph.add_graph_documents(graph_documents)

print(f"Graph updated with {len(graph_documents)} documents.")

Operational Trade-offs

Building a knowledge graph isn't free, and you need to be mindful of the overhead:

• Latency: Extracting entities using an LLM during ingestion is slow. I usually run this as an asynchronous background job rather than in the user-facing request path.
• Schema Drift: If your raw data changes frequently, your graph can become cluttered with stale nodes. I implement a "soft-delete" or versioning mechanism on nodes to ensure we don't query deprecated relationships.
• Cost: Running LLM passes over every document to extract triples adds up. If you have a massive corpus, consider using a smaller, specialized model for extraction or only processing high-value documents.

Debugging Tips

When my graph queries return empty results, I check these three things first:

1. Cypher Syntax: Use the Neo4j Browser to manually run the query generated by your LangChain agent. If it fails there, the issue is the query logic, not the LLM.
2. Node Labels: Ensure your LLMGraphTransformer is using consistent labels. Sometimes the model might infer "User" while another document uses "Account." I force a schema constraint on the LLM prompt to prevent this.
3. Relationship Direction: LLMs sometimes interpret relationships backwards (e.g., (A)-[:DEPENDS_ON]->(B) instead of (B)-[:USED_BY]->(A)). If your retrieval is failing, check if your Cypher query is ignoring relationship direction by using undirected patterns like (a)-[:DEPENDS_ON]-(b).

By combining the fuzzy matching of vector search with the deterministic traversal of Neo4j, you bridge the gap between "finding similar words" and "understanding domain logic." Start small—map out a single module or feature set before scaling to your entire knowledge base.