Core Philosophy

The basic approach is to layer different detection methods instead of relying on just one. Here's the thinking behind it:

1. Multiple Detection Layers: Fast pattern matching catches obvious stuff, statistical analysis spots behavioral changes, machine learning finds complex patterns, and external monitoring adds another perspective.
2. Context Matters: Rather than just looking at individual requests, the engine keeps track of what's normal for each IP address and URL path. So it can tell the difference between a legitimate user having a busy day versus someone probing your site.
3. Performance Considerations: Security tools shouldn't slow down your systems. The code uses modern C++ features and memory optimization to keep resource usage reasonable.
4. Operational Reality: Includes monitoring, error recovery, configuration reloading, and other features you actually need when running this in production.

Key Features

Multi-Tiered Detection System

The detection works in four tiers, each with different strengths:

• Tier 1 (Pattern Matching): Quick checks for known bad things
  • Rate limiting violations (brute force attempts, scraping)
  • Suspicious string patterns using Aho-Corasick algorithm
  • User-Agent analysis - missing UAs, known bad bots, old browsers, UA switching
  • Threat intelligence feeds for known malicious IPs
• Tier 2 (Statistical Analysis): Looks for deviations from normal behavior
  • Z-Score Analysis: How far current metrics are from historical averages
  • Behavior Change Detection: When trusted IPs suddenly act differently
  • Seasonal Patterns: Adapts to normal traffic cycles (like higher usage during business hours)
• Tier 3 (Machine Learning): Uses trained models for complex pattern recognition
  • ONNX Runtime: Runs Python-trained models directly in C++
  • Model Updates: Can swap in new models without restarting
  • Feature Engineering: About 50 different calculated features for analysis
• Tier 4 (Prometheus Integration): Advanced monitoring through PromQL
  • Custom Metrics: Define detection rules using Prometheus query language
  • Historical Data: Uses Prometheus time-series for trend analysis
  • Fault Tolerance: Handles Prometheus connectivity issues gracefully

Production Features

• Persistent State: Learned behavior patterns survive restarts
• Flexible Alerting: Sends alerts to multiple places at once
  • Console output for humans
  • JSON format for SIEM systems
  • Syslog integration
  • HTTP webhooks for Slack or custom dashboards
• Operational Stuff:
  • Configuration via config.ini file
  • Hot config reloading (no downtime)
  • Graceful shutdown handling
  • Built-in metrics and monitoring
  • Memory management with automatic cleanup

Performance & Memory stuff

• Memory Optimizations:
  • Custom hash tables optimized for IP lookups
  • Object pooling and string interning to reduce allocations
  • Hibernation of inactive state when memory gets tight
  • SIMD optimizations for string matching where possible
• Threading:
  • Lock-free data structures in hot paths
  • Work-stealing queues for better load distribution
  • Thread affinity tuning (though this is still experimental)
  • Double-buffered updates so reads don't block

Architecture Overview

How it processes logs

Each log entry goes through this pipeline:

Log Input → Parsing → Analysis & Context → Rule Checking → Alerting
     ↓         ↓           ↓               ↓           ↓
  File/stdin  LogEntry   AnalyzedEvent   RuleEngine  AlertManager
  MongoDB     Parser     +Features       +Scoring    +Dispatchers
              Pluggable  +Context        +Tiers      +Throttling

The main stages

1. Reading & Parsing (src/main.cpp, src/io/log_readers/)

   • Reads from files, stdin, or MongoDB
   • Different parsers for nginx, Apache, JSON, etc
   • Turns raw text into structured LogEntry objects

2. Analysis Engine (src/analysis/)

   • The core piece that adds context to each log entry
   • Creates AnalyzedEvent with 50+ calculated features
   • Keeps track of stats and windows for every IP, path, session
   • Has memory optimizations with cleanup when things get tight

3. Feature Manager (src/models/)

   • Converts the analyzed event into ML feature vectors
   • Handles normalization and scaling
   • Supports both training mode and inference

4. Rule Engine (src/detection/)

   • Runs the analyzed event through all the detection rules
   • Processes tiers in order of computational cost
   • Tracks metrics and performance for each rule

5. Alert Manager (src/core/)

   • Formats alerts with investigation context
   • Throttles duplicate alerts to avoid spam
   • Sends to multiple destinations with different formats

Code organization

The codebase is split into logical modules:

Runtime Controls

When the engine is running, you can control it with these signals:

Configuration Templates

There are a few pre-made configs to get you started:

# For development work - more logging and debugging
cp config_templates/development.ini config.ini

# Production setup - stable and secure defaults
cp config_templates/production.ini config.ini

# High throughput - tuned for lots of logs per second
cp config_templates/high_performance.ini config.ini

Configuration System

The configuration uses INI format with hot-reloading and validation.

Basic structure

Here's how the config is organized:

# Main settings
log_source_type = file|mongodb|stdin
log_input_path = /path/to/logs
alerts_to_stdout = true
alerts_to_file = true

# Each detection tier can be enabled/disabled
[Tier1]
enabled = true
max_requests_per_ip_in_window = 1000
suspicious_path_substrings = ../,/etc/passwd,sqlmap

[Tier2]
enabled = true
z_score_threshold = 3.5
min_samples_for_z_score = 30

[Tier3]
enabled = false
model_path = models/isolation_forest.onnx
anomaly_score_threshold = 0.6

[Tier4]
enabled = false
prometheus_url = http://localhost:9090
evaluation_interval_seconds = 60

# Other features
[MemoryManagement]
enabled = true
max_memory_usage_mb = 1024
enable_object_pooling = true

[PerformanceMonitoring]
enabled = true
enable_load_shedding = true
max_cpu_usage_percent = 80.0

[DynamicLearning]
enabled = true
confidence_threshold = 0.95
seasonal_detection_sensitivity = 0.8

Reloading config

You can change configuration without restarting:

# Send reload signal
kill -HUP <pid>

# Or if running in terminal
# Press Ctrl+R

The system checks the changes before applying them and logs any validation errors.

Environment-specific configs

Different environments need different settings:

Development (config_templates/development.ini):

• More debugging output and logging
• Lower thresholds for easier testing
• Verbose error reporting
• Fewer retry attempts for faster feedback

Production (config_templates/production.ini):

• Optimized for stability
• Conservative performance settings
• Full error handling
• Security-focused defaults

High-Performance (config_templates/high_performance.ini):

• Tuned for processing lots of logs
• Higher memory limits
• Aggressive performance settings
• Minimal logging overhead

Detection Tiers Explained

Here's how the four-tier detection system works:

Tier 1: Pattern Matching

Fast rule-based checks for known attack patterns:

Rate limiting & brute force detection

// Example: detecting brute force login attempts
if (failed_login_count > config.max_failed_logins_per_ip) {
    create_alert(TIER1_HEURISTIC, "Brute force attack detected",
                 HIGH_SEVERITY, "Block IP immediately");
}

Pattern matching

• Aho-Corasick Algorithm: Searches for multiple suspicious patterns efficiently
• Path Analysis: Looks for directory traversal (../), config access (/etc/passwd)
• User-Agent Analysis: Identifies bots, scanners, and outdated browsers
• Threat Intel: Checks against external IP reputation feeds

Implementation details

• Location: src/detection/rule_engine.cpp (functions: check_requests_per_ip_rule, check_suspicious_string_rules)
• Key Files: src/utils/aho_corasick.hpp, src/io/threat_intel/intel_manager.hpp
• Performance: Usually under 1ms per event

Tier 2: Statistical Analysis

Builds behavioral baselines and detects statistical anomalies:

Z-Score calculation

// Example: detecting unusual request volume for an IP
double z_score = (current_requests - mean_requests) / std_deviation;
if (z_score > config.z_score_threshold) {
    create_alert(TIER2_STATISTICAL, "Request volume anomaly",
                 MEDIUM_SEVERITY, "Investigate unusual activity pattern");
}

Features it tracks

• Request Patterns: Volume, timing, frequency distributions
• Error Rates: HTTP 4xx/5xx response patterns
• Content Access: HTML vs asset request ratio