🎮 JokerNet: AI-Powered Balatro Automation System

Transform your Balatro gameplay with JokerNet 🤖✨

JokerNet represents a foundational framework for AI-powered game automation that can be extended far beyond Balatro. This section outlines potential enhancements and broader applications that demonstrate the scalability and versatility of the current multiagent architecture. 🤖✨

The system integrates Azure OpenAI for intelligent reasoning, Docker for scalable deployment, and custom mods for enhanced game interaction, making it a comprehensive solution for automated gaming.

Built with state-of-the-art AI frameworks like LangChain and LangGraph, this project exemplifies expertise in AI agent orchestration, full-stack development, and containerized gaming environments. As a software engineer specializing in AI and automation, I've crafted JokerNet to demonstrate innovative approaches to game AI - from sophisticated multiagent coordination to real-time visual analysis and precise control simulation.

📋 Table of Contents

• 🚀 Key Features
• 🏗️ System Architecture
• 🤖 Multiagent System with LangGraph
• 🌐 Streamlit Web Interface
• 🐳 Docker Environment: Balatro with Enhanced Automation
• ⚠️ Mouse Controller Status
• 📦 Installation and Setup
• 📁 Project Structure
• 🔧 Configuration
• 🚀 Future Work & Extensions
• 📄 License

🚀 Key Features

🎯 AI-Powered Gameplay	🎮 Dual Control Methods	👁️ Real-Time Vision
Multiagent orchestration using LangGraph for coordinated strategic decisions	Gamepad control (primary) + Mouse control (under development)	Computer vision-powered game state recognition and dynamic planning
Learn more →	See interface →	Explore architecture →

🌐 Modern Web UI	🐳 Containerized	🔧 Custom Mods
Responsive Streamlit interface for configuration and real-time monitoring	Fully containerized Balatro environment with noVNC remote access	Enhanced game experience with auto-start and automation features
View interface →	Setup guide →	Mod details →

• 🔗 MCP Integration: Model Context Protocol server for seamless AI tool integration
• 🌐 REST API: Comprehensive HTTP endpoints for programmatic control
• 🧠 Azure OpenAI Integration: Advanced reasoning capabilities for strategic gameplay
• ⚡ Performance Optimized: GPU support, model caching, and low-latency controls

🏗️ System Architecture

JokerNet operates through a sophisticated multi-layered architecture designed for maximum scalability and modularity:

graph TB
    subgraph "User Interface Layer"
        A[Streamlit Interface<br/>Port 8501]
    end

    subgraph "AI Orchestration Layer"
        C[Multiagent System<br/>LangGraph]
    end

    subgraph "🐳 Docker Container<br/>Complete Game Environment"
        subgraph "API Services"
            F[REST API<br/>Port 8000]
            G[MCP Server<br/>Port 8001]
        end

        subgraph "Game Runtime"
            B[noVNC<br/>Port 6080]
            D[Balatro Game<br/>Love2D Engine]
            E[Custom Mods<br/>BalatroLogger]
        end

        subgraph "Display System"
            H[Xvfb<br/>Virtual Display]
            I[x11vnc<br/>VNC Server]
        end

        B --> I
        I --> H
        H --> D
        F --> D
        G --> D
        E --> D
    end

    A --> F
    A --> B
    A --> C
    C --> A
    C --> G

    style A fill:#e1f5fe
    style C fill:#f3e5f5
    style D fill:#e8f5e8
    style F fill:#fff3e0

Architecture Highlights:

• 🔄 Multi-layered Design: Clear separation between external UI, AI orchestration, and containerized game environment
• 🐳 Docker Encapsulation: All game execution components (display, APIs, game engine, mods) are contained within a single Docker image for easy deployment and portability
• 📡 Service Mesh: Streamlit interfaces with API, VNC, and multiagent system independently
• 🛡️ Fault Tolerance: Isolated components with graceful error handling
• ⚡ Performance: Optimized for responsive game automation and AI processing
• 🔗 Dual Control Paths: Both REST API and MCP server provide game control capabilities
• 🔄 Game Agnostic Design: With minimal modifications, the same Docker environment can support different games by swapping the game engine and mods
• 🔗 Bidirectional Communication: Multiagent system communicates directly with both Streamlit interface and MCP server for seamless coordination

For detailed agent orchestration, see Multiagent System

🤖 Multiagent System with LangGraph

At the heart of JokerNet lies a sophisticated multiagent system built using LangGraph, a powerful framework for orchestrating complex AI workflows. This system intelligently decomposes the challenging task of autonomous Balatro gameplay into coordinated, specialized agents that collaborate seamlessly.

🎭 Agent Roles and Coordination

The multiagent architecture features three specialized agents, each with distinct responsibilities:

Agent	Role	Specialization	Key Functions
🧠 Planner Agent	Strategic Director	High-level decision making	Task decomposition, strategic planning
👁️ Visualizer Agent	Vision Specialist	Computer vision analysis	Screenshot processing, game state extraction
⚙️ Worker Agent	Execution Expert	Action implementation	Gamepad control (primary), mouse interaction (under development)

📊 Visual Workflow Diagram

---
config:
  flowchart:
    curve: linear
---
graph TD;
	__start__([<p>__start__</p>]):::first
	planner_visualizer(planner_visualizer)
	planner(planner)
	worker_visualizer(worker_visualizer)
	worker(worker)
	tool(tool)
	output(output)
	__end__([<p>__end__</p>]):::last
	__start__ --> planner_visualizer;
	planner -. &nbsp;end&nbsp; .-> output;
	planner -.-> worker_visualizer;
	planner_visualizer --> planner;
	tool --> worker_visualizer;
	worker -.-> planner;
	worker -.-> tool;
	worker_visualizer --> worker;
	output --> __end__;
	classDef default fill:#f2f0ff,line-height:1.2
	classDef first fill-opacity:0
	classDef last fill:#bfb6fc

🔍 Workflow Breakdown:

1. 🚀 START Node: Initial user input processing and system initialization
2. 🧠 Planner-Visualizer Node: Strategic analysis with visual context capture
3. 📋 Planner Node: High-level task decomposition and strategic planning
4. 🔀 Conditional Routing: Intelligent decision branching based on planner actions
5. 👁️ Worker-Visualizer Node: Context-aware screenshot capture for execution
6. ⚙️ Worker Node: Precise action planning and tool orchestration
7. 🎮 Tool Node: Direct game control execution (gamepad actions - mouse under development)
8. 🔄 Conditional Worker Routing: Adaptive loop control with success/failure handling
9. 📤 Output Node: Final result formatting and user response delivery

🛡️ Advanced Flow Control Features:

• ⏱️ Max Step Limits: Prevents infinite loops (5 planner steps, 3 worker steps)
• 💾 State Persistence: Maintains conversation history and game state across iterations
• 🛟 Error Handling: Graceful degradation and intelligent completion detection
• 🔧 Tool Integration: Exclusive MCP server integration for game control (no direct API usage)

This architecture enables complex decision-making while ensuring system reliability and user control. For the web interface that interacts with these agents, see Streamlit Interface

🌐 Streamlit Web Interface

JokerNet features a modern, responsive Streamlit application that provides an intuitive and powerful interface for configuration, monitoring, and interaction with the AI agents.

✨ Main Features

Feature	Description	Benefits
📺 Live Game View	Embedded noVNC viewer for game monitoring	Visual feedback and control
⚙️ Agent Configuration	Direct API calls for game and agent settings	Flexible automation strategies
💬 Chat Interface	Natural language interaction with AI agents	Intuitive user experience
🎯 Run Configuration	Deck selection, stake adjustment via API	Customized gameplay scenarios
📊 Progress Monitoring	Live updates on agent actions and game state	Transparent automation process

📸 Interface Screenshots

🏠 Main Dashboard

⚙️ Agent Configuration

---

🃏 Game Configuration

---

🎮 Game Control Panel

🐳 Docker Environment: Balatro with Enhanced Automation

JokerNet includes a comprehensive Docker setup that containerizes the entire Balatro gaming environment, complete with custom mods, API servers, and remote access capabilities.

🏛️ Container Architecture

The Docker environment provides a fully isolated and optimized gaming ecosystem with all services running within a single container:

graph TB
    subgraph "🐳 Docker Container<br/>Internal Architecture"
        subgraph "API Services"
            F[REST API<br/>Port 8000]
            G[MCP Server<br/>Port 8001]
        end

        subgraph "Game Runtime"
            B[noVNC<br/>Port 6080]
            D[Balatro Game<br/>Love2D Engine]
            E[Custom Mods<br/>BalatroLogger]
        end

        subgraph "Display System"
            H[Xvfb<br/>Virtual Display]
            I[x11vnc<br/>VNC Server]
        end

        B --> I
        I --> H
        H --> D
        F --> D
        G --> D
        E --> D
    end

    style D fill:#e8f5e8
    style F fill:#fff3e0

🔧 Internal Service Flow:

• 🌐 External Access: noVNC (Port 6080) provides web-based game interface, while VNC (Port 5900) offers native client access
• 🚀 API Layer: REST API (Port 8000) handles HTTP requests, MCP Server (Port 8001) manages AI agent integration
• 🎮 Game Control: Both API services communicate directly with the Balatro game engine for programmatic control
• 🖥️ Display Pipeline: Xvfb creates virtual display → x11vnc serves VNC → noVNC provides web proxy
• 🔧 Mod Integration: Custom mods enhance game functionality and enable automation features

🐳 Docker Container Benefits:

• 🎮 Complete Game Environment: Everything needed to run and interact with the game is encapsulated in a single container
• 🔧 Easy Deployment: One-command deployment with all dependencies and s