Arbigent(Arbiter-Agent): An AI Agent Testing Framework for Modern Applications
Zero to AI agent testing in minutes. Arbigent's intuitive UI and powerful code interface make it accessible to everyone, while its scenario breakdown feature ensures scalability for even the most complex tasks.
[!WARNING]
There seems to be a spam account posing as Arbigent, but the account is not related to me. The creator's accounts are https://x.com/_takahirom_ and https://x.com/new_runnable .
Screenshot
Demo movie
https://github.com/user-attachments/assets/ec582760-5d6a-4ee3-8067-87cb2b673c8d
Motivation
Make AI Agent Testing Practical for Modern Applications
Traditional UI testing often relies on brittle methods that are easily disrupted by even minor UI changes. A/B tests, updated tutorials, unexpected dialogs, dynamic advertising, or ever-changing user-generated content can cause tests to fail.
AI agents emerged as a solution, but testing with AI agents also presents challenges. AI agents often don't work as intended; for example, the agents might open other apps or click on the wrong button due to the complexity of the task.
To address these challenges, I created Arbigent, an AI agent testing framework that can break down complex tasks into smaller, dependent scenarios. By decomposing tasks, Arbigent enables more predictable and scalable testing of AI agents in modern applications.
Customizable for Various AI Providers, OSes, Form Factors, etc.
I believe many AI Agent testing frameworks will emerge in the future. However, widespread adoption might be delayed due to limitations in customization. For instance:
- Limited AI Provider Support: Frameworks might be locked to specific AI providers, excluding those used internally by companies.
- Slow OS Adoption: Support for different operating systems (like iOS and Android) could lag.
- Delayed Form Factor Support: Expanding to form factors beyond phones, such as Android TV, might take considerable time.
To address these issues, I aimed to create a framework that empowers users with extensive customization capabilities. Inspired by OkHttp's interceptor pattern, Arbigent provides interfaces for flexible customization, allowing users to adapt the framework to their specific needs, such as those listed above.
Easy Integration into Development Workflows
Furthermore, I wanted to make Arbigent accessible to QA engineers by offering a user-friendly UI. This allows for scenario creation within the UI and seamless test execution via the code interface.
Key Feature Breakdown
I. Core Functionality & Design
- Complex Task Management:
- Scenario Dependencies: Breaks down complex goals into smaller, manageable scenarios that depend on each other (e.g., login -> search).
- Orchestration: Acts as a mediator, managing the execution flow of AI agents across multiple, interconnected scenarios.
- Hybrid Development Workflow:
- UI-Driven Scenario Creation: Allows non-programmers (e.g., QA engineers) to visually design test scenarios through a user-friendly interface.
- Code-Based Execution: Enables software engineers to execute the saved scenarios programmatically (YAML files), allowing for integration with existing testing infrastructure.
II. Cross-Platform & Device Support
- Multi-Platform Compatibility:
- Mobile & TV: Supports testing on iOS, Android, Web, and TV interfaces.
- D-Pad Navigation: Handles TV interfaces that rely on D-pad navigation.
III. AI Optimization & Efficiency
- Enhanced AI Understanding:
- UI Tree Optimization: Simplifies and filters the UI tree to improve AI comprehension and performance.
- Accessibility-Independent: Provides annotated screenshots to assist AI in understanding UIs that lack accessibility information.
- Cost Savings:
- Open Source: Free to use, modify, and distribute, eliminating licensing costs.
- Efficient Model Usage: Defaults to
gpt-4.1 for improved capabilities; also supports cost-effective models like gpt-4o-mini.
IV. Robustness & Reliability
- Double Check with AI-Powered Image Assertion: Integrates Roborazzi's feature to verify AI decisions using image-based prompts and allows the AI to re-evaluate if needed.
- Stuck Screen Detection: Identifies and recovers from situations where the AI agent gets stuck on the same screen, prompting it to reconsider its actions.
V. Advanced Features & Customization
- Flexible Code Interface:
- Custom Hooks: Offers a code interface for adding custom initialization and cleanup methods, providing greater control over scenario execution.
- Maestro YAML Integration:
- Pre-defined Test Flows: Execute existing Maestro YAML test flows as initialization methods within Arbigent scenarios.
- Reusable Test Components: Leverage your existing Maestro test automation scripts to set up complex application states before AI agent testing.
- Example Use Cases:
- Run login flows using Maestro YAML before AI agent testing
- Set up specific application states (e.g., user onboarding completion)
- Execute complex setup sequences that require precise timing
- Integrate existing Maestro test assets into AI agent testing workflows
- Model Context Protocol (MCP) Support:
- Introduced initial support for MCP, enabling Arbigent to leverage external tools and services defined via MCP servers. This significantly extends testing capabilities beyond direct UI interaction.
- You can configure MCP servers using a JSON string in the Project Settings.
- Example MCP Use Cases:
- Install and launch applications
- Check server logs (e.g., user behavior) using external tools
- Retrieve debug logs
- Interact with various other custom tools and services
- MCP Server Control:
- Project-level defaults: Use the
enabled field in MCP JSON to disable servers by default:
{
"mcpServers": {
"filesystem": { "command": "npx", "args": ["..."] },
"github": { "command": "npx", "args": ["..."], "enabled": false }
}
}
- Scenario-level overrides: Override project defaults per scenario using
mcpOptions:
scenarios:
- id: "my-scenario"
goal: "Test with specific MCP servers"
mcpOptions:
mcpServerOptions:
- name: "github"
enabled: true # Override: enable for this scenario
- name: "filesystem"
enabled: false # Override: disable for this scenario
- Servers without overrides use project defaults. If no
enabled field exists, the server is enabled by default.
- AI Hints - App-Provided Context:
- Apps can provide domain-specific hints to help the AI understand the current screen better.
VI. Community & Open Source
- Open Source Nature:
- Free & Open: Freely available for use, modification, and distribution.
- Community Driven: Welcomes contributions from the community to enhance and expand the framework.
Arbigent's Strengths and Weaknesses Based on SMURF
I categorized automated testing frameworks into five levels using the SMURF framework. Here's how Arbigent stacks up:
- Speed (1/5): Arbigent's speed is currently limited by the underlying AI technology and the need to interact with the application's UI in real-time. This makes it slower than traditional unit or integration tests.
- We have introduced some mechanisms to address this:
- Tests can be parallelized using the
--shard option to speed up execution.
- AI result caching can be utilized when the UI tree and goal are identical, which is configurable in the project settings.
- Maintainability (4/5): Arbigent excels in maintainability. The underlying AI model can adapt to minor UI changes, minimizing the need to rewrite tests for every small update, thus reducing maintenance effort. You can write tests in natural language (e.g., "Complete the tutorial"), making them resilient to UI changes. The task decomposition feature also reduces duplication, further enhancing maintainability. Maintenance can be done by non-engineers, thanks to the natural language interface.
- Utilization (1/5): Arbigent requires both device resources (emulators or physical devices) and AI resources, which can be costly. (AI cost can be around $0.005 per step and $0.02 per task when using GPT-4o.)
- Reliability (3/5): Arbigent has several features to improve reliability. It automatically waits during loading screens, handles unexpected dialogs, and even attempts self-correction. However, external factors like emulator flakiness can still impact reliability.
- Recently I found Arbigent has retry feature and can execute the scenario from the beginning. But, even without retry, Arbigent works fine without failures thanks to the flexibility of AI.
- Fidelity (5/5): Arbigent provides high fidelity by testing on real or emulated devices with the actual application. It can even assess aspects that were previously difficult to test, such as verifying video playback by checking for visual changes on the screen.
I believe that many of its current limitations, such as speed, maintainability, utilization, and reliability, will be addressed as AI technology continues to evolve. The need for extensive prompt engineering will likely diminish as AI models become more capable.
How to Use
Installation
Install the Arbigent UI binary from the Release page.
Installation for macOS Users
If you encounter security warnings when opening the app:
Refer to Apple's guide on opening apps from unidentified developers.
This Open Anyway button is available for about an hour after you try to open the app.
<img
src="http