Loading repository data…
OpenAgentHQ / repository
Local-first evaluation framework for RAG systems and AI Agents. 18+ metrics, CLI + SDK, framework-agnostic. The pytest of AI evaluation.
The open-source evaluation framework for RAG systems and AI Agents.
Local-first. Framework-agnostic. Developer-friendly.
Getting Started · Documentation · Contributing
Evaluating RAG systems shouldn't require a PhD or a cloud account. OpenAgent Eval brings pytest-level simplicity to AI evaluation — run from your terminal, get actionable insights, and ship with confidence.
pip install openagent-eval
For development:
git clone https://github.com/OpenAgentHQ/openagent-eval.git
cd openagent-eval
uv sync
oaeval init --interactive
oaeval validate config.yaml
oaeval run config.yaml
oaeval report latest
| Feature | Description |
|---|---|
| CLI + SDK | Use via command line or import as a Python library |
| Beautiful Reports | Terminal, Markdown, HTML, and JSON output formats |
| Failure Analysis | Identify why evaluations fail, not just that they failed |
| Corpus Health Auditor | Detect contradictions, staleness, and duplicates before evaluation |
| LLM-as-Judge Metrics | NLI-based scoring for faithfulness and relevancy |
| Component Diagnosis | Blame attribution — retrieval vs generation vs chunking |
| Synthetic Test Data | Auto-generate test cases from your knowledge base |
LLM Providers: OpenAI · Anthropic · Google Gemini · Groq · OpenRouter · Ollama
Retriever Providers: Chroma · Qdrant · Pinecone · Weaviate · FAISS · pgvector · Elasticsearch · BM25
| Command | Description |
|---|---|
oaeval init | Create configuration file (interactive wizard) |
oaeval run <config> | Run evaluation pipeline |
oaeval report <id> | View evaluation reports |
oaeval compare <a> <b> | Compare two experiments |
oaeval list | List previous evaluations |
oaeval validate <config> | Validate configuration |
oaeval doctor | Check environment and dependencies |
oaeval audit --corpus <path> | Audit corpus health |
oaeval diagnose --report <id> | Diagnose failures and attribute blame |
oaeval synth --corpus <path> | Generate synthetic test cases |
from openagent_eval.core import Engine
from openagent_eval.config import load_config
config = load_config("config.yaml")
engine = Engine(config)
report = await engine.run(dataset)
print(report.summary)
openagent-eval/
├── openagent_eval/
│ ├── cli/ # CLI commands (Typer)
│ ├── config/ # Configuration system (Pydantic)
│ ├── core/ # Core orchestration engine
│ ├── metrics/ # 18+ evaluation metrics
│ ├── providers/ # LLM & Retriever adapters
│ ├── corpus/ # Corpus Health Auditor
│ ├── diagnosis/ # Component Diagnosis
│ ├── synthesis/ # Synthetic Test Data
│ ├── reports/ # Report generators
│ └── plugins/ # Plugin system
├── tests/ # Test suite
├── docs/ # Documentation
└── examples/ # Tutorials and examples
We welcome contributions of all kinds. Whether you're fixing a bug, adding a feature, or improving documentation — we'd love your help.
git checkout -b feature/amazing-feature)git commit -m 'Add amazing feature')git push origin feature/amazing-feature)See CONTRIBUTING.md for detailed guidelines.
Licensed under the Apache License, Version 2.0. See LICENSE for details.
Built with care by the OpenAgent community.
Selected from shared topics, language and repository description—not editorial ratings.
Sammyco921 /
NextMind is a modular local AI agent framework built on open-source LLMs (Ollama), designed for structured task planning, tool execution, and self-evaluation through a transparent, inspectable reasoning loop.