🔧 AI Prompt Failure Diagnosis Agent

An intelligent Agentic AI system that analyzes, diagnoses, and improves ineffective prompts used with Large Language Models (LLMs). Instead of merely generating new responses, this agent focuses on understanding why a prompt failed and how to fix it — acting as an AI debugging assistant for prompt engineering.

---

🎯 Problem Statement

Traditional prompt engineering is trial-and-error. When an LLM produces poor output, users often:

• Struggle to identify what went wrong
• Make random modifications without understanding root causes
• Waste time iterating without systematic improvements
• Can't explain why changes work or don't work

This agent solves that problem by providing:

• ✅ Systematic failure classification
• ✅ Root cause analysis with explanations
• ✅ Evidence-based prompt improvements
• ✅ Transparent, teachable debugging process

---

💡 Purpose and Use Cases

Primary Purpose

Identify root causes of poor LLM outputs and provide explainable, actionable improvements.

Key Use Cases

Use Case	Description	Benefit
🐛 Debugging Ambiguous Prompts	Identify unclear instructions causing inconsistent outputs	Reliable, predictable results
🏭 Production Reliability	Improve prompt quality in production AI systems	Reduced hallucinations and errors
📚 Team Training	Teach prompt engineering best practices	Better prompts across organization
🔄 Workflow Evaluation	Assess prompt quality in agent pipelines	Optimized multi-step workflows
🎯 Hallucination Reduction	Identify and fix prompts causing false information	More trustworthy AI outputs

Who Should Use This?

• AI Engineers - Debugging production prompts
• Prompt Engineers - Learning and improving prompt design
• Product Teams - Ensuring consistent LLM behavior
• Researchers - Analyzing prompt effectiveness
• Developers - Building reliable LLM applications

---

✨ Features

🧠 Intelligent Analysis Capabilities

1. Multi-Step Agentic Reasoning

The agent performs structured, goal-driven analysis:

Input → Quality Analysis → Failure Classification → 
Root Cause Diagnosis → Prompt Improvement → Explanation

2. 8 Failure Categories

Automatically identifies these common prompt failures:

Category	Description	Example
🔴 Ambiguity	Unclear or vague instructions	"Write something about AI"
🔴 Missing Constraints	Lacks necessary boundaries	"Create content" (no length/tone)
🔴 Missing Examples	No guidance on expected output	No "for example" demonstrations
🔴 Overloaded Instructions	Too many conflicting objectives	"Be brief and comprehensive"
🔴 Poor Role Definition	Unclear context or persona	No "You are a..." statement
🔴 Formatting Issues	Output structure not specified	No format requirements
🔴 Context Loss	Missing background information	No relevant context provided
🔴 Conflicting Objectives	Contradictory instructions	"Formal but casual"

3. Quality Metrics Dashboard

Rule-based analysis provides:

• Token count and instruction density
• Presence of clear instructions
• Examples and constraints detection
• Output format specification check
• Overall quality score (0-100)

4. Root Cause Analysis

Deep technical explanation of:

• How the LLM interpreted (or misinterpreted) instructions
• What context or constraints were missing
• Why actual output differs from expectations
• Specific prompt engineering principles violated

5. Improved Prompt Generation

Creates better prompts with:

• Clear, specific instructions
• Relevant examples
• Explicit constraints
• Structured format
• Proper role/context definition
• Listed key changes and reasoning

6. Explainable AI

Every recommendation includes:

• What was wrong
• What was changed
• Why the new prompt will work better
• Specific improvements made

---

🏗️ System Architecture

Agentic Workflow

┌─────────────────────────────────────────────────────────────┐
│                        USER INPUT                           │
│  • Original Prompt (that failed)                            │
│  • Expected Output (what you wanted)                        │
│  • Actual Output (what you got)                             │
└────────────────────┬────────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────────────┐
│              NODE 1: QUALITY ANALYSIS                       │
│  Rule-Based Checks:                                         │
│  • Token count & density                                    │
│  • Instruction clarity                                      │
│  • Examples present?                                        │
│  • Constraints defined?                                     │
│  • Output format specified?                                 │
│  → Quality Score: 0-100                                     │
└────────────────────┬────────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────────────┐
│         NODE 2: FAILURE CLASSIFICATION (LLM)                │
│  AI-Powered Analysis:                                       │
│  • Identifies failure categories                            │
│  • Classifies multiple issues                               │
│  • Provides brief reasoning                                 │
│  → Output: List of failure types                            │
└────────────────────┬────────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────────────┐
│          NODE 3: ROOT CAUSE DIAGNOSIS (LLM)                 │
│  Deep Technical Analysis:                                   │
│  • Why LLM misinterpreted instructions                      │
│  • Missing context identification                           │
│  • Gap analysis (expected vs actual)                        │
│  • Principle violations                                     │
│  → Output: Detailed explanation                             │
└────────────────────┬────────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────────────┐
│        NODE 4: IMPROVED PROMPT GENERATION (LLM)             │
│  Evidence-Based Improvement:                                │
│  • Addresses identified failures                            │
│  • Adds clarity & structure                                 │
│  • Includes examples & constraints                          │
│  • Specifies format & role                                  │
│  → Output: Better prompt + reasoning                        │
└────────────────────┬────────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────────────┐
│                    OUTPUT TO USER                           │
│  • Quality metrics & score                                  │
│  • Failure categories identified                            │
│  • Root cause analysis                                      │
│  • Improved prompt                                          │
│  • Key changes with explanations                            │
└─────────────────────────────────────────────────────────────┘

State-Driven Execution

The agent uses LangGraph for orchestrated state management:

State Flow:
├── analyze_quality (Node 1)
│   ├── Success → classify_failure
│   └── Error → error_handler
│
├── classify_failure (Node 2)
│   ├── Success → diagnose_root_cause
│   └── Error → error_handler
│
├── diagnose_root_cause (Node 3)
│   ├── Success → generate_improved_prompt
│   └── Error → error_handler
│
├── generate_improved_prompt (Node 4) → END
│
└── error_handler → END

---

🛠️ Technology Stack

Component	Technology	Version	Purpose
Orchestration	LangGraph	Latest	Agent workflow and state management
LLM Provider	Groq	API	Fast inference with LLaMA 3.3 70B
LLM Model	LLaMA 3.3 70B Versatile	-	Reasoning, classification, improvement
Framework	LangChain	Latest	LLM integration and message handling
UI	Gradio	6.0+	Interactive web interface
Language	Python	3.8+	Implementation

Why These Technologies?

Groq + LLaMA 3.3

• ⚡ Lightning-fast inference (up to 10x faster than standard APIs)
• 🧠 Excellent reasoning capabilities
• 💰 Free tier available
• 🎯 Perfect for multi-step agent workflows

LangGraph

• 🔄 Stateful agent execution
• 🎯 Clear node-based architecture
• ✅ Built-in error handling
• 📊 Easy to debug and visualize

Gradio

• 🎨 Beautiful, professional UI out-of-the-box
• 🚀 Quick to deploy
• 📱 Mobile-responsive
• 🔗 Easy sharing capabilities

---

🚀 Installation

Prerequisites

• Python 3.8 or higher
• pip package manager
• Internet connection for API calls

Step 1: Clone the Repository

git clone https://github.com/HarshitWaldia/AI-Prompt-Failure-Diagnosis-Agent.git
cd prompt-diagnosis-agent

Step 2: Install Dependencies

Option A: Using pip

pip install langchain-groq langgraph gradio

Option B: Using requirements.txt

pip install -r requirements.txt

requirements.txt contents:

langchain-groq>=0.1.0
langgraph>=0.2.0
gradio>=6.0.0

Step 3: Get Your Groq API Key

Free Groq API Key Setup

1. Visit Groq Console
2. Sign up for a free account (GitHub/Google sign-in available)
3. Navigate to API Keys section
4. Click Create API Key
5. Copy your key (starts with gsk_)

Free Tier Limits:

• 14,400 requests per day
• 30 requests per minute
• More than enough for personal use

Step 4: Configure API Key

Option 1: Environment Variable (Recommended)

# Linux/Mac
export GROQ_API_KEY="your_groq_api_key_here"

# Windows (Command Prompt)
set GROQ_API_KEY=your_groq_api_key_here

# Windows (PowerShell)
$env:GROQ_API_KEY="your_groq_api_key_here"

Option 2: Enter in UI

• Launch the app and paste your key in the interface
• Key is stored for the session only

---

📖 Usage

Running the Application

python prompt_diagnosis_agent.py

The application will start and open at: http://localhost:7861

Step-by-Step Guide

1. Enter Your API Key

Paste your Groq API key in the configuration section

2. Provide Failed Prompt I