1. Client-Side User Interface: A web-based client-side user interface allows you to quickly explore and test this Agent template.
2. Orchestrator: The orchestrator allows for a dynamic, adaptable workflow with multiple orchestration options including utilizing Semantic Kernel's group chat orchestration with an intent routing agent (based on this template) as the manager agent to moderate and route the conversation, LLM function calling, CLU for intent identification only and CQA for returning exact pre-set answers only, etc. Both intent routing agent and LLM function calling options leverages CLU and CQA to provide high-quality intent identficiation and exact question-answering. See the routing strategies section down below for more details between these options.
3. Conversational Language Understanding (CLU): CLU enables you to define the top intents to ensure response quality. Whether completing a task or addressing specific customer needs, CLU provides a mechanism to ensure the agent accurately understands and executes the process of handling predefined intents. You can update the top intents as necessary to accommodate evolving business needs. Check out the CLU product documents for more details.
4. Custom Question Answering (CQA): CQA allows you to create and manage predefined QA pairs to deliver precise responses. CQA can respond consistently, improving reliability, particularly for high-stakes or regulatory-sensitive conversations. You can update the predefined QA pairs as needed to match your growing business needs. Check out the CQA product documents for more details.
5. PII Detection and Redaction (PII): Protecting user privacy is a top priority. Azure AI Language’s Personally Identifiable Information (PII) can identify and redact sensitive information before sending to LLM for processing. Check out the PII product documents for more details.
6. Large Language Model with Retrieval-Augmented Generation (LLM with RAG) to Handle Everything Else: In this template, we are showcasing a RAG solution using Azure AI Search to handle missed intents or user queries on lower-priority topics. This RAG solution can be replaced with your existing one.
7. Template Configuration for "Plug-and-Play": The template is designed to allow you to easily swap, add, or remove components to tailor to your specific needs. Whether you want to add custom intents, adjust fallback mechanisms, or incorporate additional data sources, the modular nature of this template makes it simple to configure.

Benefits

Azure AI Language already offers two services: Conversational Language Understanding (CLU) and Custom Question Answering (CQA). CLU analyzes user inputs to extract intents and entities. CQA uses pre-defined question-answer pairs or a pre-configured knowledgebase to answer user questions.

How CLU can help

A common issue with intent prediction in conversational AI is the misclassification of user intents and inaccurate entity identification. This can lead to poor user experience due to inaccurately invoked AI Agents or custom actions for intent fulfillment.

CLU can help by performing easy-to-configure and intuitive model training to map utterances to newly added pre-defined intents with high confidence. The recognition is through natural language, not hard-coded for natural interaction with users. This ensures that the system can handle inputs gracefully, provide more accurate responses, and achieve fast and low-cost intent classification.

By leveraging CLU, users can enhance their conversational AI solutions, improve intent predication accuracy, making it a valuable agent/workflow routing solution.

How CQA can help

A typical RAG solution allows users to chat with an AI agent and obtain grounded responses. Chat messages are sent directly to AOAI, where a specified model (e.g. GPT-4o) processes each message and creates a response. This is beneficial when customers have their own grounding data (e.g. product manuals, company information for Q/A). They would set up an Azure AI Search index to query their grounding data, and preprocess user chat messages by fetching relevant grounding data and passing it to the AOAI model downstream. Because the model now "knows" the grounding data to base its response around, user chats are met with contextual responses, improving the chat experience.

However, issues with RAG solutions (DSATs, or dissatisfactory examples) are hard to address. It is difficult to debug or update grounding data to fix inaccurate "grounded" responses. Further, this process can be expensive and time-consuming

Azure AI Language CQA can help address these issues and expand the functionality of existing RAG chat solutions.

Agent Architecture

This project includes a UnifiedConversationOrchestrator class that unifies the call to different routing strategies, which can intelligently route user input to intent routing agent, CLU-only, CQA-only, etc., based on the specified router type.

The intent routing agent is implemented with Semantic Kernel's group chat orchestration. It uses CLU and CQA as tools to predict intents based on your custom intents defined in CLU and answer FAQs if the corresponding question-answer pairs are created in CQA. In this orchestration, the intent routing agent takes the manager agent role to moderate and route the user queries to other appropriate agents as needed based on the results from CLU and CQA.

There is also fallback functionality when any of the following occurs: neither runtime is called, API call failed, confidence threshold not met, CLU did not recognize an intent, or CQA failed to answer the question. This fallback functionality can be configured to be any function. In this user-story, fallback will be the original RAG solution. The orchestrator object takes a string message as input, and outputs a dictionary object containing information regarding what runtime was called, relevant outputs, was fallback called, etc.

When combined with an existing RAG solution, adding a UnifiedConversationOrchestrator can help in the following ways:

• Use CQA to quick fix and override incorrect answers from RAG solution.
• Extended chat functionality based on recognized intents/entities using CLU.
• Consistent fallback to original chat functionality with RAG. Further, users can provide their own business logic to call based on CLU results (e.g. with an OrderStatus intent and OrderId entity, user can include business logic to query a database to check the order status).

The container instance demo included with this project showcases the following chat experience:

• User inputs chat dialog.
• AOAI node preprocesses by breaking input into separate utterances.
• Orchestrator routes each utterance to either an intent routing agent (with CLU and CQA), LLM function calls, directly fallback RAG, etc., based on your routing strategy value assigned to router_type in server.py (all valid routing strategy values are listed in the "Routing strategies" section below).
• If an intent defined in CLU was predicted, call extended business agent/logic based on intent/entities.
• Agent summarizes response (what business action was performed, provide answer to question, provide grounded response).

Use Case

Consider the following real-world example: Contoso Outdoors, a fictional retail company, has an existing RAG chat solution using AOAI. Their grounding data is composed of product manuals of the outdoor gear they sell. Because of this, users can easily ask the AI chat questions regarding Contoso Outdoors products (e.g. What tents do you sell?) and obtain grounded, contextual, and accurate responses.

However, if a user asks questions about the company's return policy, or something misclassified to grounding data, the RAG chat will not be able to respond accurately, as the grounding data does not contain any information regarding a return policy. It can be expensive and time consuming to update the grounding data to address this. Further, if a user asks a question about their online order status, even with updates of grounding data, RAG is not able to respond effectively here, as information is dynamic.

Incorporating CLU/CQA (through an intent routing agent or LLM function calls) using a UnifiedConversationOrchestrator solves these problems. Contoso Outdoors would set up a CQA model that can answer extended questions (e.g. their return policy), and set up a CLU model that can identify online order actions (e.g. checking the status of an order). Now, both of these DSATs are resolved, and Contoso Outdoors still maintains their existing RAG chat functionality, as UnifiedConversationOrchestrator falls back to the original RAG chat if CLU/CQA are not fit to respond to the user chat.

This displays the "better together" story when using Azure AI Language with Azure AI Agent Service and/or Azure OpenAI to provide a deterministic conversational agent experience. Learn more about related offerings from this blog post: [Announcing Azure AI Language new features to accelerate your agent development](https://te