Building Enterprise RAG on Azure

Series: Enterprise GenAI & RAG Architecture — Part 3 of 5

Why Azure is the Natural Home for Enterprise RAG

If your organization runs on Microsoft — SharePoint, Teams, Azure AD, Microsoft 365 — then Azure is your natural RAG platform. Every data source your employees use daily connects natively, without custom connectors or complex authentication flows.

Azure also provides something no other cloud currently matches: Azure AI Search with hybrid search — combining traditional BM25 keyword matching, vector similarity, and a semantic ranker in a single query. For enterprise content (policies, documentation, technical specs), this hybrid approach consistently outperforms pure vector search.

Azure AI Search’s semantic ranker uses a cross-encoder model to re-rank results after initial retrieval — dramatically improving precision for complex, multi-concept queries.

Azure Services at a Glance

Requirement	Azure Service	Notes
LLM	Azure OpenAI — GPT-4o	Latest model, 128k context window
Embeddings	text-embedding-3-small	1536 dimensions, best cost/quality ratio
Vector Database	Azure AI Search	Hybrid: BM25 + Vector + Semantic Ranker
Storage	Azure Blob Storage	Source documents staged here
Compute / Workflow	Azure Functions (Python)	Serverless, event-driven processing
CI/CD	Azure DevOps (ADO)	Pipelines, Repos, Boards — all integrated
Monitoring	Application Insights	Latency, errors, usage telemetry
Secrets Management	Azure Key Vault	API keys, connection strings, tokens

The Azure RAG Architecture — Step by Step

Data Ingestion Flow

Documents uploaded to Azure Blob Storage trigger an Azure Function automatically
The Function extracts text using PyMuPDF (PDFs), python-docx (Word), or Graph API (SharePoint)
Text is cleaned, normalized, and split into 512-token chunks with 50-token overlap
Each chunk is sent to Azure OpenAI text-embedding-3-small — returns a 1536-dim vector
Chunk text + vector + metadata (source, page, date) is upserted into Azure AI Search index

Query Flow (Every User Question)

User submits a question through the RAG application (web app, Teams bot, or API)
Question is embedded using the same text-embedding-3-small model
Azure AI Search runs a hybrid query: BM25 keyword match + vector similarity + semantic reranker
Top 5 most relevant chunks are returned with their source metadata
Chunks are injected into a structured prompt and sent to Azure OpenAI GPT-4o
GPT-4o generates a grounded answer — constrained to the retrieved context only
Answer is returned to the user with source citations

Always instruct LLM to answer ONLY from the provided context and to say ‘I don’t have that information’ when the context doesn’t contain the answer. This keeps responses grounded in verified source material and reduces unsupported AI-generated content.

Azure AI Search — Why Hybrid Mode Is Critical

Pure vector search is powerful, but it has a known weakness: exact keyword matching. If a user types an exact product code like ‘PRD-2024-XA7’, vector search may not surface the right document because the embedding captures semantic meaning — not exact strings. Azure AI Search solves this with hybrid mode:

Search Mode	Strengths	Weaknesses
BM25 Keyword	Exact term matching, product codes, IDs	No semantic understanding
Vector Search	Semantic similarity, paraphrasing, intent	Poor exact string matching
Hybrid (BM25 + Vector)	Best of both worlds — semantic + exact	Slightly higher latency
+ Semantic Ranker	Re-ranks results with deep learning cross-encoder	Additional cost per query

For enterprise use cases — especially HR policies, compliance documents, and technical specs — always enable hybrid mode with the semantic ranker.

Azure DevOps CI/CD Pipeline for RAG

A production RAG system needs automated testing on every code change. Here is the complete Azure DevOps pipeline:

Pipeline Stage	What Runs	Pass Criteria
Stage 1: Unit Tests	PyTest — chunking logic, extractor functions, prompt templates	All tests green
Stage 2: Embedding Validation	Verify 1536 dimensions, similarity scores > 0.80 for related chunks	Dimensions correct, scores pass
Stage 3: Integration Tests	Azure AI Search connectivity, Azure OpenAI API health	All connections healthy
Stage 4: RAGAS Evaluation	Context Precision, Context Recall, Faithfulness, Answer Relevance	All metrics ≥ threshold
Stage 5: Prompt Regression	PromptFoo — test 50+ golden Q&A pairs against baseline	No degradation detected
Stage 6: Security Tests	Prompt injection, PII leakage, OWASP LLM Top 10	Zero critical findings
Stage 7: Deploy	Blue/Green to Azure Container Apps or App Service	Smoke tests pass

RAGAS Thresholds for the Azure Pipeline

RAGAS Metric	Minimum Threshold	What It Measures
Context Precision	≥ 0.80	Are retrieved chunks actually relevant to the question?
Context Recall	≥ 0.75	Were all necessary facts retrieved from the knowledge base?
Faithfulness	≥ 0.85	Is the LLM answer grounded in context only? (Hallucination score)
Answer Relevance	≥ 0.80	Does the answer actually address what was asked?

If any RAGAS metric falls below threshold, the pipeline fails and the deployment is blocked. This is your automated quality gate for every release.

Security Considerations on Azure

Azure provides a mature security stack for RAG systems:

Azure Key Vault — store all API keys, connection strings, secrets — never in code or config files
Azure AD (Entra ID) — use Managed Identity for Function-to-Service authentication — no credentials needed
Azure AI Search — role-based access control via Azure RBAC — restrict who can query which indexes
Private Endpoints — keep all service communication inside your VNet — no public internet exposure
Azure Defender for AI — detect anomalous query patterns, prompt injection attempts

Managed Identity over API Keys: Configure Azure Functions to authenticate to Azure OpenAI and Azure AI Search via Managed Identity. This eliminates credential rotation risk entirely.

➡ In Part 4, we compare building RAG on AWS Bedrock vs Oracle Cloud Infrastructure — two very different approaches with distinct trade-offs for enterprise teams.