Natural Language Q&A in Optimizely CMS Using Azure OpenAI and AI Search

In Part 2, we integrated Azure AI Search with Azure Personalizer to build a smarter, user-focused experience in Optimizely CMS. We used ServiceAPI to send CMS content to Azure AI Search. In Part 3, we’ll take things a step further: enabling users to ask natural language questions and get AI-powered answers. But here’s the twist: instead of using ServiceAPI, this time we’ll explore how to query indexed content directly using Azure AI Search’s REST API.

Why Natural Language Q&A?

Sometimes users don’t know the exact keywords. They ask things like:

“How do I return a product?” “Can I book a demo?”

With OpenAI on Azure and the semantic capabilities of Azure AI Search, your website can now understand those queries and provide helpful, contextual responses.

Architecture Overview

1. CMS content is indexed by Azure AI Search (via ServiceAPI or crawler).
2. The search index is enriched with semantic settings.
3. A user types a natural language query.
4. Your backend uses Azure Search’s Semantic Search + Azure OpenAI (chat/completion API) to return a contextual answer.

Using Azure Search API to Extract Content

Revolutionize Your Business With Generative AI

From product design and software development to virtual agents, content creation, and reporting, GenAI is transforming business. Our AI experts help you unlock GenAI’s full potential and drive growth.

Let’s Get Started

You can directly access indexed content using the Azure Search REST API. Here’s an example of how to fetch top 5 results:

public async Task<List<string>> SearchAzureContentAsync(string query)
{
    var searchServiceName = "<your-search-service-name>";
    var indexName = "<your-index-name>";
    var apiKey = "<your-api-key>";

    using var client = new HttpClient();
    client.DefaultRequestHeaders.Add("api-key", apiKey);

    var url = $"https://{searchServiceName}.search.windows.net/indexes/{indexName}/docs?api-version=2021-04-30-Preview&search={query}&$top=5";
    var result = await client.GetStringAsync(url);
    
    var json = JsonDocument.Parse(result);
    var docs = new List<string>();

    foreach (var item in json.RootElement.GetProperty("value").EnumerateArray())
    {
        docs.Add(item.GetProperty("content").GetString());
    }

    return docs;
}

Generate Answer Using Azure OpenAI

Once we retrieve relevant documents, we’ll pass them into Azure OpenAI to generate a contextual answer:

public async Task<string> AskOpenAiAsync(string question, List<string> context)
{
    var prompt = $"""
You are a helpful assistant. Based on the following content, answer the question:

{string.Join("\n\n", context)}

Question: {question}
Answer:
""";

    var openAiKey = "<your-openai-key>";
    var endpoint = "https://<your-openai-endpoint>.openai.azure.com/openai/deployments/<deployment-name>/completions?api-version=2022-12-01";

    var payload = new
    {
        prompt = prompt,
        temperature = 0.7,
        max_tokens = 200
    };

    using var client = new HttpClient();
    client.DefaultRequestHeaders.Add("api-key", openAiKey);
    var response = await client.PostAsync(endpoint, new StringContent(JsonSerializer.Serialize(payload), Encoding.UTF8, "application/json"));

    var result = JsonDocument.Parse(await response.Content.ReadAsStringAsync());
    return result.RootElement.GetProperty("choices")[0].GetProperty("text").GetString();
}

Integrate with Optimizely CMS

You can create a controller like this:

public class QnAController : Controller
{
    [HttpPost]
    public async Task<ActionResult> Ask(string question)
    {
        var docs = await SearchAzureContentAsync(question);
        var answer = await AskOpenAiAsync(question, docs);
        return Json(new { answer });
    }
}

And on your view:

<form method="post" id="qnaForm">
  <input type="text" name="question" placeholder="Ask a question..." />
  <button type="submit">Ask</button>
</form>
<div id="answer"></div>

<script>
  $('#qnaForm').submit(function(e) {
    e.preventDefault();
    const question = $('input[name=question]').val();
    fetch('/QnA/Ask', {
      method: 'POST',
      headers: {'Content-Type': 'application/json'},
      body: JSON.stringify({ question })
    })
    .then(r => r.json())
    .then(data => $('#answer').text(data.answer));
  });
</script>

Summary

This wraps up the final puzzle piece: letting users speak freely with your Optimizely site, while AI interprets and responds in real-time. From content indexing to re-ranking to full-on Q&A, your CMS is now intelligent, conversational, and user-first. Want to see this in action? Stay tuned for the sample repo and video walkthrough!

The blog is also published here Natural Language Q&A in Optimizely CMS Using Azure OpenAI and AI Search