Multi-Agent-Medical-Assistant

---

[!IMPORTANT]
📋 Version Updates from v2.0 to v2.1 and further:

1. Document Processing Upgrade: Unstructured.io has been replaced with Docling for document parsing and extraction of text, tables, and images to be embedded.
2. Enhanced RAG References: Links to source documents and reference images present in reranked retrieved chunks stored in local storage are added to the bottom of the RAG responses.

To use Unstructured.io based solution, refer release - v2.0.

📚 Table of Contents

• Overview
• Demo
• Technical Flow Chart
• Key Features
• Tech Stack
• Installation and Setup
  • Using Docker
  • Manual Installation
• Usage
• Contributions
• License
• Citing
• Contact

---

📌 Overview

The Multi-Agent Medical Assistant is an AI-powered chatbot designed to assist with medical diagnosis, research, and patient interactions.

🚀 Powered by Multi-Agent Intelligence, this system integrates:

• 🤖 Large Language Models (LLMs)
• 🖼️ Computer Vision Models for medical imaging analysis
• 📚 Retrieval-Augmented Generation (RAG) leveraging vector databases
• 🌐 Real-time Web Search for up-to-date medical insights
• 👨‍⚕️ Human-in-the-Loop Validation to verify AI-based medical image diagnoses

What You’ll Learn from This Project 📖

🔹 👨‍💻 Multi-Agent Orchestration with structured graph workflows
🔹 🔍 Advanced RAG Techniques – hybrid retrieval, semantic chunking, and vector search
🔹 ⚡ Confidence-Based Routing & Agent-to-Agent Handoff
🔹 🔒 Scalable, Production-Ready AI with Modularized Code & Robust Exception Handling

📂 For learners: Check out agents/README.md for a detailed breakdown of the agentic workflow! 🎯

---

💫 Demo

https://github.com/user-attachments/assets/d27d4a2e-1c7d-45e2-bbc5-b3d95ccd5b35

If you like what you see and would want to support the project's developer, you can ! :)

📂 For an even more detailed demo video: Check out Multi-Agent-Medical-Assistant-v1.9. 📽️

---

🛡️ Technical Flow Chart

---

✨ Key Features

• 🤖 Multi-Agent Architecture : Specialized agents working in harmony to handle diagnosis, information retrieval, reasoning, and more

• 🔍 Advanced Agentic RAG Retrieval System :

  • Docling based parsing to extract text, tables, and images from PDFs.
  • Embedding markdown formatted text, tables and LLM based image summaries.
  • LLM based semantic chunking with structural boundary awareness.
  • LLM based query expansion with related medical domain terms.
  • Qdrant hybrid search combining BM25 sparse keyword search along with dense embedding vector search.
  • HuggingFace Cross-Encoder based reranking of retrieved document chunks for accurate LLM reponses.
  • Input-output guardrails to ensure safe and relevant responses.
  • Links to source documents and images present in reference document chunks provided with reponse.
  • Confidence-based agent-to-agent handoff between RAG and Web Search to prevent hallucinations.

• 🏥 Medical Imaging Analysis

  • Brain Tumor Detection (TBD)
  • Chest X-ray Disease Classification
  • Skin Lesion Segmentation

• 🌐 Real-time Research Integration : Web search agent that retrieves the latest medical research papers and findings

• 📊 Confidence-Based Verification : Log probability analysis ensures high accuracy in medical recommendations

• 🎙️ Voice Interaction Capabilities : Seamless speech-to-text and text-to-speech powered by Eleven Labs API

• 👩‍⚕️ Expert Oversight System : Human-in-the-loop verification by medical professionals before finalizing outputs

• ⚔️ Input & Output Guardrails : Ensures safe, unbiased, and reliable medical responses while filtering out harmful or misleading content

• 💻 Intuitive User Interface : Designed for healthcare professionals with minimal technical expertise

[!NOTE]
Upcoming features:

1. Brain Tumor Medical Computer Vision model integration.
2. Open to suggestions and contributions.

---

🛠️ Technology Stack

Component	Technologies
🔹 Backend Framework	FastAPI
🔹 Agent Orchestration	LangGraph
🔹 Document Parsing	Docling
🔹 Knowledge Storage	Qdrant Vector Database
🔹 Medical Imaging	Computer Vision Models
	• Brain Tumor: Object Detection (PyTorch)
	• Chest X-ray: Image Classification (PyTorch)
	• Skin Lesion: Semantic Segmentation (PyTorch)
🔹 Guardrails	LangChain
🔹 Speech Processing	Eleven Labs API
🔹 Frontend	HTML, CSS, JavaScript
🔹 Deployment	Docker, GitHub Actions CI/CD

---

🚀 Installation & Setup

📌 Option 1: Using Docker

Prerequisites:

• Docker installed on your system
• API keys for the required services

1️⃣ Clone the Repository

git clone https://github.com/souvikmajumder26/Multi-Agent-Medical-Assistant.git
cd Multi-Agent-Medical-Assistant

2️⃣ Create Environment File

• Create a .env file in the root directory and add the following API keys:

[!NOTE]
You may use any llm and embedding model of your choice...

1. If using Azure OpenAI, no modification required.
2. If using direct OpenAI, modify the llm and embedding model definitions in the 'config.py' and provide appropriate env variables.
3. If using local models, appropriate code changes might be required throughout the codebase especially in 'agents'.

[!WARNING]
Ensure the API keys in the .env file are correct and have the necessary permissions. No trailing whitespaces after variable names.

# LLM Configuration (Azure Open AI - gpt-4o used in development)
# If using any other LLM API key or local LLM, appropriate code modification is required
deployment_name= 
model_name=gpt-4o
azure_endpoint=
openai_api_key=
openai_api_version=

# Embedding Model Configuration (Azure Open AI - text-embedding-ada-002 used in development)
# If using any other embedding model, appropriate code modification is required
embedding_deployment_name=
embedding_model_name=text-embedding-ada-002
embedding_azure_endpoint=
embedding_openai_api_key=
embedding_openai_api_version=

# Speech API Key (Free credits available with new Eleven Labs Account)
ELEVEN_LABS_API_KEY=

# Web Search API Key (Free credits available with new Tavily Account)
TAVILY_API_KEY=

# Hugging Face Token - using reranker model "ms-marco-TinyBERT-L-6"
HUGGINGFACE_TOKEN=

# (OPTIONAL) If using Qdrant server version, local does not require API key
QDRANT_URL=
QDRANT_API_KEY=

3️⃣ Build the Docker Imag