Directory Structure

swarm-pim/
├── backend/
│   ├── app/
│   │   ├── __init__.py
│   │   ├── agents.py       # Multi-agent logic and LLM tools
│   │   ├── cache.py        # Qdrant semantic memory layer
│   │   ├── config.py       # Environment variable resolution
│   │   ├── database.py     # DuckDB connection and Arrow initialization
│   │   ├── main.py         # REST endpoints and routing
│   │   └── schemas.py      # Pydantic validation frames
│   ├── Containerfile       # Multi-stage Python build instructions
│   └── requirements.txt    # Frozen backend dependencies
├── frontend/
│   ├── app/                # Next.js pages and routing state
│   ├── package.json        # Node dependency manifest
│   └── Containerfile       # Multi-stage Node/Next production build
├── data/                   # Persistent host volume mount mapping
│   ├── data.duckdb         # Persistent storage target
│   └── qdrant_storage/     # Persistent vector indices
└── podman-compose.yaml     # Service definitions and environment bindings

Configuration

The system requires specific environment configurations for execution.

Create an .env file inside the backend/ directory using the structure defined below:

# API Gateway Keys
GEMINI_API_KEY=your_production_gemini_credential
GROQ_API_KEY=your_production_grok_credential

# Engine Runtime Flags
DATABASE_PATH=/app/data/data.duckdb
QDRANT_HOST=embedded

Installation & Deployment

Prerequisites

Ensure your host machine has the Podman Engine and the Podman-Compose toolchain installed.

If running on Windows, verify that:

• WSL2 is active
• Container storage is mapped appropriately
• Sufficient disk space exists for .vhdx virtual machine files

Clean Local Build

To execute a full multi-stage compilation and initialization of the service mesh, navigate to the root directory and run:

podman compose -f podman-compose.yaml up --build -d

Note: The --build flag is strictly required when dependencies are modified in requirements.txt or package.json to overwrite cached filesystem layers.

Runtime Commands

Service Status Logs

To view unified application logs across the gateway and frontend clusters in real-time:

podman compose -f podman-compose.yaml logs -f

Stopping Services

To pause execution without wiping container contexts:

podman compose -f podman-compose.yaml stop

Tearing Down Services

To dismantle the local network bridge and terminate the container processes while leaving physical database state intact on the host storage volume:

podman compose -f podman-compose.yaml down

Deep Infrastructure Verification

To access the internal runtime shell of the running backend container for live debugging or file structure verification:

podman exec -it swarmpim-backend ls -la /app/data

Service Endpoints

Once successfully booted, the services are bound to the host network interface:

Technical Deep Dive: Architectural Knowledge Brief

This section provides a granular, production-level breakdown of the system mechanics for developers analyzing the engineering decisions behind the SwarmPIM repository.

1. Zero-Copy Ingestion: DuckDB, Polars, and Apache Arrow

When dealing with data transfers between storage engines and web runtimes, the conventional approach involves pulling data into a backend model, serializing it into standard JSON strings, and transmitting it over network sockets. In high-throughput environments, this incurs heavy CPU costs due to memory copying and string processing overhead.

SwarmPIM mitigates this via an analytical pipeline using DuckDB and Polars.

The In-Memory Bridge

DuckDB natively supports export data operations in the Apache Arrow format. Apache Arrow defines a language-independent column-oriented memory layout.

Zero-Copy Execution

When the Next.js client requests data from /api/v1/catalog, FastAPI invokes DuckDB's internal execution engine to fetch the structured rows.

Instead of converting rows into native Python dicts, the query returns an Arrow record batch stream.

Polars Ingestion

Polars initializes a DataFrame directly on top of these Arrow memory pointers using pyarrow.

Because both utilities share an identical specifications layer for column data, the memory allocation is reused without duplication.

Performance Impact

This pipeline reduces memory pressure and serialization bottlenecks.

It enables fast database querying and transmission speeds even when handling large batch sizes of deeply structured schema properties.

2. Multi-Agent Orchestration and Resilient LLM Routing

The multi-agent system does not rely on complex, unconstrained chain-of-thought prompts.

It follows a modular design with clear validation separation.

[Incoming Catalog Payload]
            │
            ▼
┌───────────────────────┐
│     Drafter Agent     │ ◄─── Validates structural extraction types
└───────────┬───────────┘
            │
            ▼
┌───────────────────────┐
│    Validator Agent    │ ◄─── Executes schema checks & constraints
└───────────┬───────────┘
            │
    ┌───────┴───────┐
    ▼               ▼
[Passed]        [Failed]
    │               │
    ▼               ▼
[Save State]    [Feedback Loop to Drafter]

The Drafting Tier

The Drafter Agent receives an unorganized string payload.

It parses raw specifications, standardizes text attributes, resolves missing categories, and matches product values into structured data fields.

It outputs strict types enforced by Python validation schemas.

The Validation Tier

The Validator Agent reviews the output from the Drafter.

It runs programmatic checks against configuration parameters, validates mathematical boundaries (e.g., pricing or dimensions), and confirms that critical attributes are accurate.

If the validator identifies data gaps or contradictions, it creates an accurate error log and routes the data back to the Drafter in an automated loop.

Failover Resiliency

To prevent system crashes during upstream service interruptions, the provider client implements a layered execution loop.

3. High-Performance Cache Routing via Semantic Memory

To minimize unnecessary LLM calls, every raw data string undergoes semantic preprocessing before hitting the agent pipelines.

Vector Processing

Raw input fields are converted into vector representations using a local embedding algorithm.

Distance Evaluation

The embedding is sent to an embedded instance of Qdrant.

The vector database runs distance calculations against existing indices using cosine similarity parameters.

Cache Fault Isolation

If a vector signature falls within the matching threshold, it indicates an identical or nearly identical duplicate request.

The cache retrieves the structured results directly from the stored database index, resolving the request in milliseconds.

If the distance vector exceeds thresholds, a cache miss is logged.

The system forwards the payload to the AI swarm, and caches the final validated schema back into Qdrant for subsequent hits.

4. Container Ephemerality and Mount Boundaries

When configuring container topologies for applications deployed on Windows via WSL2, understanding the separation between execution memory and persistent storage is critical.

Stateful Volume Decoupling

Containers are designed to be ephemeral.

If a container instance is dropped, modified, or destroyed via a down operation, any stateful changes made inside its isolated filesystem layers are instantly lost.

To guarantee database persistence across builds, SwarmPIM establishes explicit boundaries using host-mounted volumes.

volumes:
  - ./data:/app/data:z

The :z flag ensures safe shared SELinux labeling permissions when mounted across varying host file formats.

Resource Profiles and Host VM Interactions

The Container State

When running:

podman compose stop

container instances are paused, dropping active CPU use to 0% while retaining container cache tables.

A down command safely deletes the container instances entirely.

The WSL2 Layer Engine

Even when containers are completely purged via down, the parent Linux instance (podman machine) remains active in system memory, holdi