NeoMind

What is NeoMind?

NeoMind is an edge-deployed AI platform that brings intelligence to IoT. It runs LLM-powered agents directly on your hardware, connecting to devices via MQTT/BLE/Webhook, automating responses through a rule engine, and visualizing everything on real-time dashboards — all without relying on cloud services.

Key idea: Talk to your devices in natural language. The AI understands your intent, queries device states, creates automation rules, and takes action autonomously.

📚 Full documentation is on the NeoMind Wiki. This README is a quick overview — visit the wiki for complete guides:

• What is NeoMind? — product overview and concepts
• Five-Minute Quick Start — get running fast
• Install & Setup — deployment, desktop app, server, Docker
• Developer Guide — API, extensions, integrations

Why NeoMind?

• Fully self-contained — Embedded MQTT broker, redb storage, no external database or broker to install
• Type-safe end-to-end — Rust backend with compile-time guarantees; agent CLI commands dispatch in-process with structured data, no fragile string parsing
• Crash-proof extensions — Extensions run in isolated processes with capability-based permissions; a misbehaving extension never takes down the server
• Cloud-optional — Works 100% offline with local LLMs (Ollama), or connect cloud models when you need more power

Features

AI-Powered Intelligence

• Natural Language Chat — Conversational interface to query and control all connected devices
• Autonomous Agents — Scheduled AI agents that monitor, analyze, and act on device data independently
• 10+ LLM Backends — Ollama, OpenAI, Anthropic, Google, xAI, Qwen, DeepSeek, GLM, MiniMax, and any OpenAI-compatible endpoint
• Memory System — Multi-tier memory (User, Knowledge, Procedures, Session) with automatic extraction and compression
• Skill System — YAML+Markdown skills that guide agent behavior for specific scenarios
• Multimodal — Image upload and visual analysis support

Device Management

• MQTT Protocol — Primary device integration with embedded broker, mTLS, and CA certificate support
• BLE Provisioning — Zero-touch device setup via Bluetooth (Tauri native + Web Bluetooth)
• HTTP/Webhook — Flexible device adapter for REST-based devices
• Auto-Discovery — Automatic device detection, type registration, and AI-assisted onboarding
• Command Queue — Send control commands to devices with parameter validation and tracking
• Custom Device Types — Define device metrics and commands via JSON type definitions

Automation

• Rule Engine — JSON-based rules with recursive conditions (comparison/range/logical) and actions (notify/execute/trigger_agent), with cooldown and for_duration debouncing
• Data Transforms — JavaScript-based data transformation for creating virtual metrics
• Scheduled Agents — Time-based and event-driven AI agent execution
• Event Bus — Pub/sub architecture for decoupled component communication

Dashboards & Visualization

• Drag-and-Drop Builder — Visual dashboard editor with responsive grid layout
• Rich Widgets — Value cards, charts, gauges, tables, VLM vision components
• Real-time Updates — WebSocket/SSE for live data streaming to dashboards
• Dashboard Sharing — Public links with expiration for sharing dashboards
• Custom Components — Build and publish your own dashboard widgets

Notification & Data Push

• 7 Notification Channels — Webhook, Email, Telegram, WeCom, DingTalk, Slack, Feishu
• Data Push — Forward telemetry data to external systems via Webhook or MQTT
• Delivery Tracking — Retry logic with exponential backoff, delivery history, and log management
• Message Deduplication — Prevent notification storms from high-frequency triggers

Platform

• Multi-Instance — Connect to and manage multiple NeoMind backends from a single interface
• Extension System — Native & WASM extensions with process isolation and capability-based permissions
• Cross-Platform Desktop — macOS, Windows, Linux native apps via Tauri
• Mobile-Friendly Web — Responsive web UI optimized for phone and tablet
• i18n — English and Chinese language support
• Dark Mode — System-aware dark/light theme
• API Key Auth — Alternative to JWT for programmatic access
• CLI Tools — Full-featured command-line interface for all operations

Ecosystem

NeoMind is a modular ecosystem with specialized repositories for each concern:

Repository	Purpose
NeoMind	Core platform (this repo) — backend, frontend, desktop app
NeoMind-Extensions	Official extension marketplace — 22 extensions: vision (YOLO/face/OCR), voice (TTS/ASR), IoT bridges (HA/Modbus/BACnet/ONVIF/OPC-UA/LoRaWAN), and more
NeoMind-DeviceTypes	Device type definitions — standardized metrics and commands for IoT hardware
NeoMind-Dashboard-Components	Dashboard widget marketplace — community-contributed React components

Available Extensions

22 official extensions across vision, voice, IoT bridges, and utilities. Browse the full list in the Extensions Marketplace.

Extension	Category	Description
Image Analyzer	Vision	YOLOv11 object detection on uploaded images (80+ COCO categories)
YOLO Video	Vision	Real-time object detection on RTSP/RTMP/HLS streams with ROI + line crossing
YOLO Device Inference	Vision	Auto YOLOv11 detection on NE301/NE101 camera feeds with smart capture
Face Recognition	Vision	ArcFace enrollment, matching, and real-time detection on device streams
OCR Device Inference	Vision	SVTR text recognition bound to device image streams
PaddleOCR-VL	Vision	High-accuracy multilingual OCR, table recognition, and key info extraction
Locate Anything	Vision	Visual grounding — object detection, phrase grounding, GUI grounding, OCR
CosyVoice 3	Voice	Fun-CosyVoice3-0.5B streaming TTS (host audio + wav synthesis)
MOSS TTS Nano	Voice	Voice-cloning TTS extension
SenseVoice ASR	Voice	Multilingual (zh/en/ja/ko/yue) speech recognition via sherpa-onnx
Edge TTS	Voice	sherpa-onnx ZipVoice cross-platform CPU streaming TTS
Voice Assistant	Voice	Full pipeline orchestrator: mic → VAD → ASR → reply → TTS → speaker
Home Assistant Bridge	IoT Bridge	Import 3000+ HA entity integrations as NeoMind devices
LoRaWAN Bridge	IoT Bridge	ChirpStack/TTN sensors with auto-discovery + payload decoding
Modbus Bridge	IoT Bridge	TCP/RTU PLCs, power meters, sensors, industrial devices
BACnet Bridge	IoT Bridge	BACnet/IP building automation discovery and control
ONVIF Bridge

Supported Devices

NE301 (Edge AI Camera) and NE101 (Sensing Camera). See NeoMind-DeviceTypes for full device type definitions.

Contribute to the Ecosystem

We welcome community contributions to grow the NeoMind ecosystem:

• Build an Extension — Create extensions for new data sources, AI models, or integrations. Follow the Extension Development Guide to get started, then submit a PR to the marketplace.
• Add a Device Type — Define metrics and commands for your IoT hardware so others can use it out of the box. Just add a JSON file.
• Create a Dashboard Widget — Build reusable React dashboard components (charts, gauges, maps, etc.) and share them with the community.

Quick Start

For the full walkthrough see the Five-Minute Guide and Install & Setup on the wiki.

Desktop App (Recommended)

Download the latest release from GitHub Releases.

Platform	Format
macOS (Apple Silicon + Intel)	.dmg
Windows	.msi / .exe
Linux	.AppImage / .deb

On first launch, a setup wizard guides you through creating an admin account, configuring your LLM backend, and connecting devices.

Server Deployment

One-line install (Linux & macOS):

curl -fsSL