Key Functionalities

• Skin Image Processing and disease classification
• Context-aware chat response
• medical information retrieval
• Dermatologist Search
• Session management
• Error handling and validation

🔗 endpoint link

Access the Medical Assistant Endpoint

Steps on How to Clone And Run the Project

1. Clonning the project

 git clone https://github.com/COMFORTINE-SIWENDE/AI-Powered-Dermatology-Assistant.git

2. Navigate to root dir, Create a new virtual environment and activate it

• In linux distribution

python3 -m venv venv
cd venv
Source venv/bin/activate
pip install -r requirements.txt

• In Window os

python -m venv venv
cd venv
\Scripts\activate
pip install -r requirements.txt

• create migrations and migrate

python manage.py makemigrations assistant

python manage.py migrate

• Run React App in new terminal

cd client
npm run dev

• Create a .env file within the endpoints dir and provide the following(shoul not be strings).For the database you can configure sqlite3 which is by default in django.
• Modify your dabase settings to use your preferred database in settings.py

AZURE_OPENAI_API_KEY=xxxx........
AZURE_OPENAI_API_ENDPOINT=xxxx........
AZURE_OPENAI_API_VERSION=.............
AZURE_AI_SEARCH_KEY=xxxx.......
AZURE_AI_SEARCH_ENDPOINT=xxxx....
DATABASE_PASSWORD=your_db_password

• Run the Django api endpoints in another terminal

cd endpoints
python manage.py runserver 8000

🧭 Project Workflow

🪄 The Autonomous Agentic System Design

🔄 Routing

• The application employs dynamic routing to seamlessly process both image-based diagnoses and text-based medical conversations through a single, unified API endpoint (/api/medical-assistant). When a request is received, the system automatically detects the input type—whether it’s an uploaded skin photo or a text query describing symptoms—and intelligently routes it to the appropriate AI model. Image inputs are analyzed by a CNN-based classifier, which generates a diagnosis along with confidence scores, while text inputs are processed by Azure OpenAI (GPT model) enhanced with LangChain and Azure AI Search to deliver accurate, context-aware medical responses. This streamlined approach ensures users can effortlessly transition between submitting visual diagnostics and asking follow-up questions, all within a single, intuitive interaction flow.

⛓️ Prompt Chaining

• Contextual follow-up:If the user asks a follow-up question (e.g., "How to treat this?"), the system chains prompts by:

  Retrieving the prior diagnosis from chat history (Azure PostgreSQL).
  

  Injecting it into a new GPT prompt: "The user has psoriasis (92% confidence). Provide a concise treatment plan using data from Azure AI Search, including topical treatments and lifestyle advice."

• Multi-modal integration:For complex queries (e.g., "Is this contagious?"), the system combines:Image analysis results (from CNN), Medical guidelines (from Azure AI Search) and Conversational context (from chat history) .

  Example chained prompt: "Based on the user's psoriasis diagnosis (image-attached), explain contagion risks and prevention steps. Cite dermatology guidelines."

🧠 Retrieval-Augmented Generation

RAG (Retrieval-Augmented Generation) enhances the AI’s responses by grounding them in up-to-date, authoritative medical knowledge—combining the reasoning of Azure OpenAI GPT with targeted data retrieval from Azure AI Search.

Knowledge Retrieval

Medical Corpus:
Azure AI Search indexes trusted sources such as clinical guidelines, research papers, or drug databases.

Contextual Queries:
When a user asks, "What’s the first-line treatment for psoriasis?", LangChain:

• Converts the question into a search query.
• Fetches relevant excerpts (e.g., "Topical corticosteroids are preferred for mild psoriasis...").

RAG transforms the AI from a chatbot into a context-aware medical assistant—bridging the gap between LLMs and clinical expertise.

Need details on fine-tuning retrievers or handling ambiguous queries?

Project Tools,Frameworks,Technologies and Libraries

• Azure Machine Learning Compute
• Azure AI Search
• Azure OpenAI
• Langchain
• Tensorflow
• Django REST
• ReactJs
• PostgreSQL

🧠 Convolution Neural Network

CNN Model Training with Azure Machine Learning Studio & VS Code Integration

• To develop our skin disease classifier, we designed a Sequential CNN with 3 convolutional blocks (Conv2D + MaxPooling2D), totaling 11 layers, and trained it using Azure Machine Learning Studio’s GPU clusters, orchestrated via VS Code on a local desktop. Here’s a breakdown of the process:

Model Architecture & Specifications

📥 Input

• Shape: (180x180x3) RGB images
• Preprocessing: Standardized for consistency across all inputs

🧱 Convolutional Blocks

• Structure: 3 sequential blocks, each containing:
  • Conv2D layer with ReLU activation
  • MaxPooling2D layer for downsampling
• Purpose: Hierarchical feature extraction from input images

📤 Output Layer

• Type: Dense (fully-connected)
• Neurons: 17 (one for each skin condition class)
• Parameters: 2,193 trainable parameters

📊 Model Summary

Training Infrastructure

Azure ML Studio GPU Clusters

• VM Type: NC-series (NC6s_v3)
• GPU: NVIDIA Tesla V100
• Features:
  • Automated job scheduling
  • Hyperparameter tracking (learning rate, batch size)
  • Metric logging (accuracy, loss)

VS Code Integration

• Connected local VS Code to Azure ML via the Azure Machine Learning Extension.

• Developed/trained the model interactively using Jupyter Notebooks( skin_disease.ipynb )

endpoint/skin_disease.ipynb

The CNN model summary and testing

Saving the generalized CNN Model

• 📊 Total Parameters: 11.97m

• ✅ Memory footprint: - model weight:45.67 MB - optimizer States:30.45

• Source: Connected to the REST API endpoint.

• Destination: Configured to route data to the dma_lakehouse.

• Transformation: Utilized Eventstream's capabilities to filter and transform incoming data before storage in lakehouse delta tables

Data Pipeline & Analytics Workflow in Fabrics

🧩 Components Overview

• Lakehouse: dma_lakehouse For storing data from REST api in delta tables
• Eventstream: For handling real-time data ingestion from the REST API.
• PySpark Notebook: For Processesing and transforming data for analysis.
• Semantic Model: For structuring the data for business intelligence.
• Power BI Dashboard: Visualizing insights and trends.

🌐 End-to-End Real-Time Data Pipeline in Microsoft Fabric

1. 🛠️ REST API Integration

• Source:Dermatology RESTful API providing JSON-formatted data related to skin disease predictions.

• Data: Includes session IDs, request types, timestamps, diagnosis details, conversation history, suggested actions, and status.

2. ⚡ Eventstream Setup

3. 🏞️ Lakehouse Storage

• Purpose: Served as the centralized repository for structured and unstructured data.

• Integration: Data from Eventstream is stored in Delta Lake format within the Lakehouse, enabling efficient querying and processing.

4. 🧪 Data Processing with PySpark

Notebook: Developed a PySpark notebook to process and analyze the ingested data.

Schema Definition: Defined a structured schema to parse complex JSON data accurately.

Transformations:

• Extracted relevant fields from nested JSON structures.
• Converted timestamps and calculated additional metrics as needed.

5. 📚 Semantic Model Creation

Model: Built the dermatology semantic - Model to structure data for analysis.

Features:

• Organized data into meaningful entities and relationships.
• Defined measures and calculated columns for insights.

6. 📈 PowerBI Analytics and Visualization

Dashboard: Developed an interactive Power BI dashboard to visualize predictions and trends.

Components:

• Disease Distribution: Donut chart showing frequency of predicted diseases.
• Confidence Analysis: Funnel chart displaying predic