How to Test a RAG System: RAGAS, Hallucination Detection, and the Complete GenAI CI/CD Pipeline

Series: Enterprise GenAI & RAG Architecture — Part 5 of 5 

Why Testing a RAG System is Different

Testing a traditional API is straightforward: given input X, expect output Y. Testing a RAG system is categorically different. You are simultaneously validating:

• Data quality — are the right documents ingested correctly?
• Chunking quality — are chunks meaningful and complete?
• Embedding quality — do similar topics score highly together?
• Retrieval accuracy — are the most relevant chunks returned?
• LLM output quality — is the answer grounded, accurate, and complete?
• Security — is the system resistant to prompt injection and PII leakage?

Each layer can fail independently. A perfect LLM with bad retrieval still produces bad answers. This is why RAG testing requires a layered, systematic approach.

The most expensive RAG failure is silent degradation — retrieval quality slowly drifts as documents change, and nobody notices until users stop trusting the system.

The RAG Testing Pyramid — 4 Layers

Borrow the concept of the traditional test pyramid and apply it to RAG. Build from the foundation up:

Layer 1: Data Quality Testing

This is the cheapest, fastest layer — and the most overlooked. Bad data causes silent failures downstream.

• Are all expected source documents present in the vector DB?
• Are there any duplicate chunks (same content stored twice)?
• Are any chunks empty, too short (< 50 tokens), or garbled text?
• Is metadata complete — source filename, page number, and ingestion date?

def test_no_empty_chunks(chunks):
    for chunk in chunks:
        assert len(chunk.text.strip()) > 50, f'Empty chunk: {chunk.id}'
        assert chunk.metadata['source'] is not None

def test_no_duplicate_chunks(chunks):
    texts = [c.text for c in chunks]
    assert len(texts) == len(set(texts)), 'Duplicate chunks detected'

def test_document_count(vector_db, expected_count):
    actual = vector_db.get_document_count()
    assert actual >= expected_count, f'Expected {expected_count}, got {actual}'

Layer 2: Embedding Quality Testing

• Embedding generated for every chunk — no nulls or errors
• Correct vector dimensions: 1536 (Azure), 1024 (AWS/OCI)
• Semantically similar chunks score above 0.80 cosine similarity
• Unrelated chunks score below 0.30 — no false positives

def test_embedding_dimensions(embedding):

    assert len(embedding) == 1536  # For Azure text-embedding-3-small

def test_semantic_similarity():

    score = cosine_similarity(

        embed('annual leave entitlement'),

        embed('how many days holiday do I get?')

    )

    assert score > 0.80, f'Similarity too low: {score}'

def test_dissimilar_chunks_score_low():

    score = cosine_similarity(

        embed('leave policy'), embed('network firewall rules')

    )

    assert score < 0.30, f'False similarity detected: {score}'

Layer 3: Retrieval Testing with RAGAS

RAGAS (Retrieval Augmented Generation Assessment) is the industry-standard framework for evaluating RAG systems. It provides metric-driven, reproducible evaluation across your entire pipeline.

Context Precision — ‘Is what we retrieved actually relevant?’

Measures the proportion of retrieved chunks that are genuinely relevant to the question.

# Example calculation:
# Retrieved 5 chunks for: 'What is the expense claim limit?'
# Relevant: chunks 1, 3, 4  (about expenses)
# Irrelevant: chunks 2, 5  (about leave policy, office hours)
# Context Precision = 3/5 = 0.60  -->  FAIL (threshold: 0.80)

Context Recall — ‘Did we retrieve everything needed?’

Measures whether all facts required to answer correctly were present in the retrieved context.

# Example calculation:
# Ground truth answer requires 4 key facts
# Retrieved context contains 3 of those facts
# Context Recall = 3/4 = 0.75  -->  PASS (threshold: 0.75)

from ragas import evaluate

from ragas.metrics import context_precision, context_recall

results = evaluate(
    dataset=test_dataset,  # questions + ground truths + retrieved contexts
    metrics=[context_precision, context_recall]
)

assert results['context_precision'] >= 0.80, 'Precision below threshold'
assert results['context_recall']    >= 0.75, 'Recall below threshold'
print(f"Precision: {results['context_precision']:.2f}")
print(f"Recall:    {results['context_recall']:.2f}")

Layer 4: LLM Response Testing — Hallucination Detection

The Four Response Quality Metrics

Hallucination in Detail

Faithfulness measures whether every claim in the LLM’s answer can be traced back to the retrieved context. A claim that cannot be traced is a hallucination.

# Context says: 'Employees are entitled to 20 days annual leave per year'
#
# BAD answer: 'Employees get 25 days plus 5 bonus days for performance'
# --> '25 days' and 'bonus days' are NOT in context
# --> Faithfulness = 0.0  -->  HALLUCINATION  -->  FAIL
#
# GOOD answer: 'According to company policy, employees receive 20 days annual leave'
# --> Every claim traces back to the context
# --> Faithfulness = 1.0  -->  PASS

Additional hallucination detection tools:

• DeepEval HallucinationMetric — standalone hallucination scorer with explanation
• PromptFoo — test known Q&A pairs, flag when answers deviate from expected output
• LangSmith — full chain tracing to see exactly what context was passed and what was generated

The Complete Enterprise CI/CD Pipeline

Every RAG system change — whether code, documents, or configuration — must go through this automated quality gate before reaching production.

The most important insight: RAGAS evaluation is NOT just a one-time quality check. Run it continuously in production. Retrieval quality drifts as your document base evolves.

QA Automation Framework — The Complete Stack

As a QA Lead or SDET on a RAG project, this is your complete toolbox:

Recommended Test Folder Structure

tests/
├── data/
│   ├── test_ingestion.py        # All source docs ingested
│   ├── test_chunking.py         # No empty or duplicate chunks
│   └── test_metadata.py         # Source, page, date correct
├── embeddings/
│   ├── test_embedding_dim.py    # Vector shape correct
│   └── test_similarity.py       # Semantic similarity scores
├── retrieval/
│   ├── test_ragas_precision.py  # Context Precision >= 0.80
│   └── test_ragas_recall.py     # Context Recall >= 0.75
├── llm/
│   ├── test_faithfulness.py     # Hallucination detection
│   ├── test_relevance.py        # Answer relevance
│   └── test_promptfoo.py        # Golden Q&A regression
└── security/
    ├── test_prompt_injection.py  # Injection blocked
    └── test_pii_leakage.py       # No PII in responses

Series Summary — 9 Principles to Remember

1. RAG is the enterprise standard. Private data + LLM = grounded, accurate answers. No retraining needed.
2. Every pipeline stage needs testing. Ingest → Chunk → Embed → Store → Retrieve → Generate — each can fail independently.
3. Chunking quality determines retrieval quality. 512 tokens + 10% overlap is your safe default. Invest time tuning this.
4. Never switch embedding models without re-indexing. It will silently break your entire retrieval layer.
5. RAGAS is non-negotiable. Context Precision + Recall measure retrieval. Faithfulness + Relevance measure LLM output.
6. CI/CD for RAG is mandatory. Every document change can silently break retrieval. Automate your quality gate.
7. Choose cloud by ecosystem. Microsoft shop → Azure. Multi-model → AWS. Oracle DB customers → OCI.
8. Security testing for LLMs is different. OWASP LLM Top 10 is your checklist. Prompt injection is your highest risk.
9. Monitoring never stops. Set up continuous RAGAS evaluation in production. Drift is silent and inevitable.

References

• RAGAS — https://docs.ragas.io
• DeepEval — https://docs.confident-ai.com
• PromptFoo — https://promptfoo.dev
• LangSmith — https://smith.langchain.com
• Azure AI Search — https://learn.microsoft.com/en-us/azure/search/
• AWS Bedrock — https://docs.aws.amazon.com/bedrock
• Oracle AI Vector Search — https://docs.oracle.com/en/database/oracle/oracle-database/23/vecse/
• OWASP LLM Top 10 — https://owasp.org/www-project-top-10-for-large-language-model-applications/