What is RAG?
RAG (Retrieval-Augmented Generation) is a technique that combines information retrieval with a Large Language Model (LLM) instead of asking an LLM to answer a question only from pretrained knowledge, RAG first retrieves relevant information from specific knowledge sources, the retrieved information is then provided to the LLM as context, which helps the LLM generate a more relevant and grounded answer.
Typical RAG flow:
User Question → Retrieve Relevant Chunks → Provide Context to LLM → Generate Answer
Why Do We Need RAG Evaluation?
Building RAG pipeline does not automatically guarantee that it will produce correct answers. There can be problems at different stages of the pipeline.
For example:
Problem 1 – Poor Retrieval
The question may be:
“What is the work-from-home policy?”
But the retriever may return chunks about leave policy instead.
The LLM now has incorrect or irrelevant context.
Problem 2 – Missing Information
The correct document may contain the answer, but the retrieval process may fail to retrieve the required chunk.
In this case, LLM may not have enough information to provide the correct answer.
Problem 3 – Unsupported Answer
The retrieved context may contain the correct information, but the LLM may generate an answer that includes information that isn’t present in the context.
This is commonly referred to as hallucination or lack of grounding.
Problem 4 – Irrelevant Answer
The retrieved context may be correct, and the answer may contain information, but it may not actually answer the user’s question.
Therefore, simply checking the final response manually isn’t enough.
We need a systematic way to evaluate different aspects of the RAG pipeline.
What is Ragas?
Ragas is an open-source framework for evaluating LLM applications, with strong support for Retrieval-Augmented Generation (RAG) pipelines.
Ragas provides evaluation metrics that help measure different aspects of a RAG system.
Instead of simply asking:
“Is this answer good?”
We can evaluate questions such as:
- Did we retrieve the right information?
- Did we retrieve enough relevant information?
- Is the generated answer supported by the retrieved context?
- Does the answer address the user’s question?
This allows us to identify where the RAG pipeline is performing well and where it needs improvement.
What We Are Going to Build
In this blog, we will use a PDF as the knowledge source. We will generate synthetic test data from the PDF using Ragas, run each generated question through the RAG pipeline, and use an LLM as a judge to evaluate the results.
We will evaluate four metrics: Answer Relevancy, Faithfulness, Context Precision, and Context Recall.
The overall flow is simple:
PDF → Load PDF → Chunk PDF → Generate Test Data using Ragas → Run each question through RAG → LLM as Judge → Evaluation
1. Install the Required Packages
First, install the libraries required for this example.
pip install -U ragas langchain langchain-community langchain-openai langchain-text-splitters pypdf faiss-cpu python-dotenv
2. Load the PDF
Let’s assume our input document is called employee_handbook.pdf.
We first load the PDF using PyPDFLoader.
3. Chunk the PDF
A PDF can contain a lot of information. We split it into smaller chunks so that the RAG application can retrieve the relevant information when a question is asked.
4. Create the RAG Retriever
For our simple example, we create embeddings for the chunks and store them in FAISS (can use any vector DB here). The retriever will use this store to find relevant chunks for each question.
5. Generate Test Data Using Ragas
This is one of the useful parts of Ragas.
Instead of manually creating questions from the PDF, Ragas can generate synthetic test data based on the document content.
For example, Ragas can generate questions such as:
• What is the leave policy?
• How many vacation days are available?
• Who approves a leave request?
This gives us a test dataset that can be used to test our RAG pipeline.
6. Run Each Question Through the RAG Pipeline
Now we take each generated question and send it to our actual RAG application.
The RAG application retrieves relevant chunks and uses an LLM to generate the answer.
For each test question, we collect three important things:
- The question
- The generated answer
- The retrieved context
7. Use an LLM as a Judge
Now comes the evaluation part.
We use LLM as a judge. Its job is not to generate the RAG answer. Its job is to evaluate the answer produced by the RAG application.
The judge looks at the question, answer, retrieved context, and reference information as required by the metric.
8. Evaluate the RAG Using Four Metrics
To evaluate the RAG pipeline, we will use four metrics that assess the quality of retrieved context and generated answers.
1. Answer Relevancy
What does it measure? Does the generated answer directly address the user’s question?
Example:
Question: How many vacation days are available?
Relevant answer: Employees receive 20 vacation days per year.
If the answer talks about unrelated employee benefits instead, the answer is less relevant.
In simple → Did the answer address the question?
2. Faithfulness
What does it measure? Is the answer supported by the retrieved context?
If the retrieved context says employees receive 20 vacation days, but the RAG answer says 30 days, the answer is not supported by the context.
In simple → Did I answer using the retrieved information?
3. Context Precision
What does it measure? Did we retrieve relevant information?
For a question about leave policy, we want the retriever to return leave-policy information rather than unrelated information such as parking or IT security.
In simple → Did I retrieve relevant information?
4. Context Recall
What does it measure? Did we retrieve all the important information needed to answer the question?
If the answer needs information A, B, and C but the retriever only finds A and B, some important information is missing.
In simple → Did I retrieve all the important information?
9. Understand the Results
Suppose we get the following scores, we can understand them simply:
Answer Relevancy = 0.91 → The answers are mostly relevant to the questions.
Faithfulness = 1.0 → The answers are mostly supported by the retrieved context.
Context Precision = 1.0 → Most retrieved information is relevant.
Context Recall = 0.5 → Some important information may not have been retrieved.
The actual pass/fail threshold should be decided based on the requirements of your application.
Conclusion
Without an evaluation framework, we may test an RAG application by asking a few questions and manually checking the answers. Ragas provides a simple way to evaluate a RAG application using measurable metrics.
References
Ragas Documentation: https://docs.ragas.io/
Ragas Test Data Generation: https://docs.ragas.io/en/stable/concepts/test_data_generation/
Ragas Evaluation: https://docs.ragas.io/en/latest/references/evaluate/
