Overview

CloudAudit is an enterprise-grade cloud storage security posture auditing framework. It automatically discovers, crawls, and analyses publicly exposed cloud storage containers — AWS S3, Google Cloud Storage, Azure Blob Storage, open directory listings — to identify secret leakage, credential exposure, compliance gaps, and misconfigurations before attackers do.

CloudAudit is designed for defensive internal use only. Every design decision prioritises safety:

• Read-only by architecture: The HTTP client exposes only GET, HEAD, and OPTIONS. PUT and DELETE do not exist.
• Ownership-gated: The engine refuses to start without explicit --confirm-ownership and --org-name flags.
• Secrets never stored raw: All pattern matches are redacted before being written to findings or transmitted to AI providers.
• No exploitation guidance: AI providers are explicitly prompted to produce only defensive remediation recommendations.

Technical Architecture

Layered Architecture

graph TD
    CLI["CLI Layer<br/>cli/main.py, cli/display.py<br/>argparse entry-point, ownership gate, interactive AI setup"]
    Config["Config Layer<br/>core/config.py, config_mgr/key_manager.py<br/>AuditConfig, Fernet-encrypted key storage, PBKDF2-SHA256 salt"]
    Scanner["Scanner Engine<br/>scanners/container_detector.py, scanners/crawler.py,<br/>scanners/secret_scanner.py, scanners/file_classifier.py,<br/>scanners/archive_extractor.py<br/>AWS S3 XML pagination, Azure XML, HTML directory crawl<br/>20+ regex patterns + Shannon entropy gate<br/>ZIP/TAR/GZ/JAR/WAR extraction (zip-slip + bomb protected)"]
    AI["AI Integration Layer<br/>ai/providers.py, ai/analyzer.py<br/>Provider Abstraction with Gemini, OpenAI, Claude, Ollama, Heuristic<br/>ProviderChain: primary → heuristic fallback<br/>Dynamic Gemini model discovery via client.models.list()<br/>No hardcoded model names — zero 404 errors"]
    Intel["Intelligence Layer<br/>intelligence/advanced.py, intelligence/risk_scorer.py, intelligence/image_meta.py<br/>High-entropy detection, duplicate detection<br/>Exposure surface mapping, composite risk scoring v2<br/>EXIF GPS/device extraction"]
    Report["Reporting Engine<br/>reports/generator.py<br/>JSON (SIEM-ready), HTML (Jinja2 stakeholder), Markdown<br/>Executive summary embedded, Severity distribution charts"]

    CLI --> Config
    Config --> Scanner
    Scanner --> AI
    AI --> Intel
    Intel --> Report

┌─────────────────────────────────────────────────────────────────────┐
│                          CLI Layer                                  │
│   cli/main.py  ·  cli/display.py                                    │
│   argparse entry-point · ownership gate · interactive AI setup      │
└───────────────────────────────┬─────────────────────────────────────┘
                                │
┌───────────────────────────────▼─────────────────────────────────────┐
│                        Config Layer                                 │
│   core/config.py  ·  config_mgr/key_manager.py                      │
│   AuditConfig · Fernet-encrypted key storage · PBKDF2-SHA256 salt   │
└───────────────────────────────┬─────────────────────────────────────┘
                                │
┌───────────────────────────────▼─────────────────────────────────────┐
│                       Scanner Engine                                │
│   scanners/container_detector.py  ·  scanners/crawler.py            │
│   scanners/secret_scanner.py  ·  scanners/file_classifier.py        │
│   scanners/archive_extractor.py                                     │
│   ─────────────────────────────────────────────────────────────     │
│   AWS S3 XML pagination · Azure XML · HTML directory crawl          │
│   20+ regex patterns + Shannon entropy gate                         │
│   ZIP/TAR/GZ/JAR/WAR extraction (zip-slip + bomb protected)         │
└───────────────────────────────┬─────────────────────────────────────┘
                                │
┌───────────────────────────────▼─────────────────────────────────────┐
│                     AI Integration Layer                            │
│   ai/providers.py  ·  ai/analyzer.py                                │
│   ─────────────────────────────────────────────────────────────     │
│   Provider Abstraction  ──►  GeminiProvider (google.genai SDK)      │
│                          ──►  OpenAICompatibleProvider (GPT-4o)     │
│                          ──►  ClaudeProvider (Anthropic)            │
│                          ──►  OllamaProvider (local)                │
│                          ──►  HeuristicProvider (always available)  │
│   ─────────────────────────────────────────────────────────────     │
│   ProviderChain: primary → heuristic fallback                       │
│   Dynamic Gemini model discovery via client.models.list()           │
│   No hardcoded model names — zero 404 errors                        │
└───────────────────────────────┬─────────────────────────────────────┘
                                │
┌───────────────────────────────▼─────────────────────────────────────┐
│                    Intelligence Layer                               │
│   intelligence/advanced.py  ·  intelligence/risk_scorer.py          │
│   intelligence/image_meta.py                                        │
│   ─────────────────────────────────────────────────────────────     │
│   High-entropy detection · Duplicate/reuse detection                │
│   Exposure surface mapping · Composite risk scoring v2              │
│   EXIF GPS/device extraction                                        │
└───────────────────────────────┬─────────────────────────────────────┘
                                │
┌───────────────────────────────▼─────────────────────────────────────┐
│                     Reporting Engine                                │
│   reports/generator.py                                              │
│   ─────────────────────────────────────────────────────────────     │
│   JSON (SIEM-ready) · HTML (Jinja2 stakeholder) · Markdown          │
│   Executive summary embedded · Severity distribution charts         │
└─────────────────────────────────────────────────────────────────────┘

AI Integration Layer — Detail

graph TD
    A["API Key"] --> B["GeminiProvider.__init__()"]
    B --> C["client.models.list()"]
    C --> D{"Filter: generateContent support?"}
    D -- "keep" --> E["Filter: not deprecated / vision-only / embedding"]
    E --> F["Sort: ultra > pro > flash > others"]
    F --> G["Store best candidate as self._model_name"]
    G --> H["GeminiProvider.complete(prompt)"]
    H --> I["client.models.generate_content"]
    I --> J{"Success?"}
    J -- "yes" --> K["AIResponse(text, provider, model, latency_ms)"]
    J -- "no" --> L["ProviderAuthError / ProviderError"]
    L --> M["ProviderChain catches → next provider → HeuristicProvider"]
    M --> N["AIResponse.ok → caller receives clean string"]
    K --> N

API Key
  │
  ▼
GeminiProvider.__init__()
  │
  ├─► client.models.list()           ← dynamic discovery, no hardcoded names
  │         │
  │         ▼
  │   Filter: generateContent support
  │   Filter: not deprecated / vision-only / embedding
  │   Sort:   ultra > pro > flash > others
  │         │
  │         ▼
  │   self._model_name = best_candidate   ← stored once, reused forever
  │
  ▼
GeminiProvider.complete(prompt)
  │
  ├─► client.models.generate_content(model=self._model_name, ...)
  │         │
  │         ├─ success → AIResponse(text, provider, model, latency_ms)
  │         │
  │         └─ failure → ProviderAuthError (401/403) or ProviderError
  │                           │
  │                           ▼
  │                    ProviderChain catches → next provider → HeuristicProvider
  │
  ▼
AIResponse.ok  →  caller receives clean string, never crashes

Model Selection Flow

graph TD
    A["client.models.list()"] --> B["Filter models: keep those with generateContent support"]
    B --> C["Filter models: exclude deprecated, vision-only, embedding"]
    C --> D["Sort by preference: ultra > pro > flash > others"]
    D --> E{"Any models left?"}
    E -- "Yes" --> F[Select first model; log 'GeminiProvider initialised with model: X']
    E -- "No" --> G["Raise ProviderError (fail loudly)"]

client.models.list()
        │
        ▼
  for model in all_models:
    │
    ├─ supported_generation_methods includes "generateContent"? → keep
    ├─ name contains "vision"/"embedding"/"deprecated"?         → skip
    └─ neither?                                                 → keep
        │
        ▼
  Sort candidates by preference: ultra > pro > flash > others
        │
        ▼
  Select candidates[0]  ──►  log "GeminiProvider initialised with model: X"
        │
        ▼
  No candidates → raise ProviderError (fail loudly, never silently)

Error Handling Flow

graph TD
    Start["AI call"] --> Outcome{"Outcome"}

    Outcome -->|ProviderAuthError| Auth["Surface immediately, no retry (bad key)"]
    Outcome -->|ProviderError| PE["ProviderChain logs warning and advances to next provider"]
    Outcome -->|Empty response| Empty["Log warning and return AIResponse(ok=False)"]
    Outcome -->|HeuristicProvider| Heur["Deterministic summary, always succeeds, never raises"]

AI call
  │
  ├─ ProviderAuthError  → surface immediately, no retry (bad key)
  │
  ├─ ProviderError      → ProviderChain logs warning, advances to next provider
  │
  ├─ Empty response     → log warning, return AIResponse(ok=False)
  │
  └─ HeuristicProvider  → deterministic summary, always succeeds, never raises