khaos

Khaos is a Kafka data generator, load testing tool, and chaos engineering CLI. Generate synthetic test data, simulate realistic producer and consumer workloads, and inject failures into your Kafka cluster. Perfect for testing Spark Streaming, Apache Flink, and Kafka Streams applications.

Table of Contents

• Use Cases
• Features
• Requirements
• Installation
• Quick Start
• Shell Completion
• CLI Reference
  • Commands Overview
  • run vs simulate
  • Cluster Modes
  • khaos cluster-up
  • khaos cluster-down
  • khaos cluster-status
  • khaos list
  • khaos validate
  • khaos run
  • khaos simulate
• Learning Stream Processing
• Available Scenarios
• Creating Custom Scenarios
  • Basic Structure
  • Topic Configuration
  • Message Schema
  • Structured Field Schemas
  • Serialization Formats
  • Producer Config
  • Consumer Config
  • Incident Primitives
  • Incident Groups
• Correlated Event Flows
• Testing Patterns
  • Duplicate Message Simulation
  • Consumer Failure Simulation
• Kafka Cluster Details
• Running with Docker
• License

Use Cases

• Kafka Data Generator: Generate fake Kafka messages with realistic schemas for testing
• Kafka Load Testing: Stress test and benchmark your Kafka cluster at high throughput
• Kafka Producer Simulator: Simulate multiple producers with configurable rates and patterns
• Stream Processing Testing: Generate test data for Apache Flink, Spark Streaming, and Kafka Streams
• Chaos Engineering: Inject broker failures, trigger rebalances, simulate consumer lag
• Kafka Consumer Testing: Test consumer group behavior, lag scenarios, and rebalancing
• Monitoring Validation: Verify Grafana dashboards and alerting rules with real traffic patterns

Features

• One-Command Setup: Spin up a 3-broker Kafka cluster with traffic in seconds
• YAML-Based Scenarios: Define traffic patterns declaratively, no code required
• Producer-Only Mode: Generate data without built-in consumers (--no-consumers)
• External Cluster Support: Connect to any Kafka cluster (self-hosted, external)
• Chaos Engineering: Built-in incident primitives (backpressure, rebalances, broker failures)
• Full Authentication: SASL/PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, SSL/TLS, mTLS
• Live Stats Display: Real-time producer/consumer metrics
• Serialization Formats: JSON, Avro, and Protobuf with Schema Registry support
• Web UI: Kafka UI at localhost:8080 for cluster inspection

Requirements

• Python 3.11+
• Docker and Docker Compose (for local cluster)

Installation

Using Homebrew (macOS/Linux)

brew install khaos

Using uv (recommended)

uv tool install khaos-cli

# Add to PATH (if not already):
uv tool update-shell

# Or manually:
export PATH="$HOME/.local/bin:$PATH"

Using pipx

pipx install khaos-cli

Using pip

pip install khaos-cli

Verify installation

khaos --version
khaos --help

From source

git clone https://github.com/aleksandarskrbic/khaos.git
cd khaos

# Option 1: Use without installing globally (recommended for development)
uv sync
uv run khaos --help

# Option 2: Install as global command
uv tool install -e .
export PATH="$HOME/.local/bin:$PATH"  # if not already in PATH
khaos --help

Development Setup

# Clone and install dependencies
git clone https://github.com/aleksandarskrbic/khaos.git
cd khaos
uv sync

# Install pre-commit hooks
uv run pre-commit install

# Run all checks before committing (lint, format, test, typecheck)
./scripts/check.sh

Quick Start

# Run a scenario (auto-starts local Kafka cluster)
khaos run traffic/high-throughput

# Press Ctrl+C to stop

---

Shell Completion

Enable tab completion for commands, options, and scenarios:

# Install completion (bash, zsh, fish, powershell)
khaos --install-completion

# Restart shell or reload config
source ~/.zshrc  # or ~/.bashrc

Then use Tab to autocomplete:

khaos cl<TAB>          # → cluster-up, cluster-down, cluster-status
khaos run <TAB>        # → shows available scenarios
khaos run --<TAB>      # → shows available options

---

CLI Reference

Commands Overview

Command	Description
khaos cluster-up	Start the 3-broker Kafka cluster
khaos cluster-down	Stop the Kafka cluster
khaos cluster-status	Show Kafka cluster status
khaos list	List available traffic scenarios
khaos validate	Validate scenario YAML definitions
khaos run	Run scenarios on local Docker cluster
khaos simulate	Run scenarios on external Kafka cluster

run vs simulate

Both commands execute the same YAML scenarios with the same traffic patterns and incident triggers. The difference is where they run:

	run	simulate
Target cluster	Local Docker (auto-managed)	Any external Kafka cluster
Docker management	Auto starts/stops cluster	No Docker interaction
Authentication	None needed	Full support (SASL, SSL, mTLS)
Broker incidents	Full support (stop_broker, start_broker)	Skipped (cannot control external brokers)
All other incidents	Full support	Full support

When to use run:

• Local development and testing
• Full chaos engineering with broker failure simulation
• Quick experiments without external dependencies

When to use simulate:

• Load testing external/self-hosted clusters
• Testing authentication configurations
• Running chaos scenarios on staging/production environments
• When you need broker incidents skipped (they're automatically skipped with a warning)

---

Cluster Modes

khaos supports two Kafka deployment modes:

Mode	Description
kraft	Default. Modern KRaft mode (no ZooKeeper) - Kafka 3.x+
zookeeper	Legacy ZooKeeper mode - for testing older deployments

Both modes run the same 3-broker cluster with identical ports and capabilities.

---

khaos cluster-up

Start the 3-broker Kafka cluster in Docker.

# Start with KRaft mode (default)
khaos cluster-up

# Start with ZooKeeper mode
khaos cluster-up --mode zookeeper
khaos cluster-up -m zookeeper

Options:

Option	Short	Default	Description
--mode	-m	kraft	Cluster mode: kraft or zookeeper

This starts:

• 3 Kafka brokers (kafka-1, kafka-2, kafka-3)
• ZooKeeper (only in zookeeper mode)
• Kafka UI at http://localhost:8080

---

khaos cluster-down

Stop the Kafka cluster.

# Stop cluster (keep data)
khaos cluster-down

# Stop cluster and remove all data volumes
khaos cluster-down --volumes
khaos cluster-down -v

Options:

Option	Short	Description
--volumes	-v	Remove data volumes

---

khaos cluster-status

Show the status of Kafka containers.

khaos cluster-status

---

khaos list

List all available traffic scenarios.

khaos list

---

khaos validate

Validate scenario YAML files for errors.

# Validate all scenarios
khaos validate

# Validate specific scenario(s)
khaos validate traffic/high-throughput
khaos validate traffic/consumer-lag traffic/hot-partition

---

khaos run

Run one or more traffic simulation scenarios on the local Docker Kafka cluster.

Auto-starts the cluster if not running. After the scenario completes, the cluster is stopped (unless --keep-cluster is specified).

khaos run SCENARIO [SCENARIO...] [OPTIONS]

Options:

Option	Short	Default	Description
--duration	-d	0	Duration in seconds (0 = run until Ctrl+C)
--keep-cluster	-k	false	Keep Kafka cluster running after scenario ends
--bootstrap-servers	-b	127.0.0.1:9092,...	Kafka bootstrap servers
--mode	-m	kraft	Cluster mode: kraft or zookeeper
--no-consumers	-	false	Disable built-in consumers (producer-only mode)

Examples:

# Run built-in scenario until Ctrl+C
khaos run traffic/high-throughput

# Run custom scenario file
khaos run ./my-scenario.yaml
khaos run /path/to/custom-scenario.yaml

# Run for 60 seconds
khaos run traffic/high-throughput --duration 60
khaos run traffic/high-throughput -d 60

# Run multiple scenarios together
khaos run traffic/hot-partition chaos/rebalance-storm

# Run multiple scenarios for 2 minutes
khaos run traffic/consumer-lag chaos/throughput-drop --duration 120

# Keep cluster running after scenario (for manual inspection)
khaos run traffic/high-throughput --keep-cluster
khaos run traffic/high-throughput -k

# Keep cluster running with duration
khaos run traffic/high-throughput -d 60 -k

# Use custom bootstrap servers (still uses local Docker cluster)
khaos run traffic/high-throughput --bootstrap-servers localhost:9092

# Run with ZooKeeper mode (instead of KRaft)
khaos run traffic/high-throughput --mode zookeeper
khaos run traffic/high-throughput -m zookeeper

# Producer-only mode (no built-in consumers)
# Useful for learning stream processing with Spark/Flink
khaos run traffic/high-throughput --no-consumers -k

---

khaos simulate

Run traffic simulation against an external Kafka cluster (self-hosted, etc.).

Unlike run, this command:

• Does NOT start/stop Docker infrastructure
• Automatically skips broker incidents (stop_broker, start_broker)
• Supports full authentication (SASL, SSL/TLS, mTLS)

khaos simulate SCENARIO [SCENARIO...] [OPTIONS]

Options:

Option	Short	Required	Default	Description
--bootstrap-servers	-b	Yes	-	Kafka bootstrap servers
--duration	-d	No	0	Duration in seconds (0 = run until Ctrl+C)
--security-protocol	-	No	PLAINTEXT	PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL
--sasl-mechanism	-	No	-	PLAIN, SCRAM-SHA-256, SCRAM-SHA-512
--sasl-username	-	No	-	SASL username
--sasl-password	-	No	-	SASL password
--ssl-ca-location	-	No	-	Path to CA certificate file
--ssl-cert-location	-	No	-	Path to client certif