Loading repository data…
Sanvith6 / repository
A production-grade, highly secure distributed AI inference platform on AWS. Provisions isolated private subnets via Terraform, automates configuration with Ansible, and implements a polyglot Python/TypeScript WebSocket RPC worker mesh behind Nginx. Includes an interactive web showcase.
[!TIP]
🌟 Interactive Portfolio Showcase
Experience a visual, interactive simulation of this entire infrastructure!
Open
SHOWCASE.htmlin any web browser to explore:
- 🌐 Interactive System Architecture Topology Map with packet-flow visualizers
- ⚡ Interactive API Completions Simulator showing complete trace logs
- 📟 Mock Shell Terminal with diagnostic command executions
- 📂 Technical Blueprint Explorer for all HCL, Ansible YAML, and systemd units
- 🩺 Real-Time Debugging Journal documenting real incident investigations
This project implements a fully automated, production-grade distributed AI inference system deployed across multiple isolated AWS EC2 instances. It combines Infrastructure-as-Code (Terraform), configuration management (Ansible), and a WebSocket-based RPC mesh (iii-engine) to expose a private, multi-worker AI inference pipeline through a public HTTPS API.
Workers written in Python and TypeScript run exclusively inside a private subnet — completely isolated from the internet — and communicate with the central engine via outbound WebSocket connections. A self-signed TLS-terminated Nginx reverse proxy sits at the public edge, rate-limiting and forwarding HTTP requests into the internal inference mesh.
Key achievement: A client can call
POST /v1/chat/completionsover HTTPS, which travels through Nginx → iii-engine → TypeScript caller-worker → Python inference-worker — all dynamically routed via WebSocket RPC — returning an OpenAI-style JSON response.
| Feature | Detail |
|---|---|
| 🏗️ Infrastructure as Code | 100% Terraform — VPC, subnets, EC2, IAM, SGs, NAT, routing |
| 🤖 Ansible Automation | Full configuration management — packages, systemd, code deployment |
| 🔒 Private Subnet Isolation | Workers have no public IPs and are unreachable from the internet |
| 🌐 NAT Instance Routing | Cost-optimized self-managed NAT (~$3.80/mo vs $32/mo managed) |
| 🔐 HTTPS API Gateway | TLS-terminated Nginx reverse proxy with security headers and rate limiting |
| 🔌 WebSocket RPC Mesh | Bidirectional outbound WS connections — workers register dynamically |
| 🧩 Polyglot Workers | Python inference worker + TypeScript caller worker communicating via RPC |
| 📦 systemd Services | All workers run as hardened systemd units with Restart=on-failure |
| 🩺 Health Endpoints | /health readiness check + /v1/chat/completions for inference validation |
| 🛡️ SSM Integration | AWS Systems Manager access on all VMs — no direct SSH port exposure required |
| 📊 IAM Least Privilege | Separate IAM roles for engine and workers with scoped policy attachments |
| ♻️ Resiliency Built-In | Exponential backoff reconnect loops + systemd auto-restart boundaries |
┌──────────────────────────────────────────────────────────────────┐
│ AWS VPC (10.0.0.0/16) │
│ │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ Public Subnet (10.0.1.0/24) │ │
│ │ │ │
Internet ──► │ │ ┌──────────────────────────────────┐ ┌──────────────┐ │ │
HTTPS :443 │ │ │ VM1 — Engine & API Gateway │ │ VM4 — NAT │ │ │
│ │ │ • Nginx (TLS, rate-limit) │ │ • ip_forward │ │ │
│ │ │ • iii-engine (:3111 local) │ │ • iptables │ │ │
│ │ │ • WS listener (:49134) │ │ MASQUERADE │ │ │
│ │ │ • Elastic Public IP │ └──────┬───────┘ │ │
│ │ └────────────┬──────────┬──────────┘ │ outbound │ │
│ └───────────────┼──────────┼────────────────────┼──────────┘ │
│ │ WS :49134│ WS :49134 │ │
│ ┌───────────────┼──────────┼────────────────────┼──────────┐ │
│ │ Private Subnet (10.0.2.0/24) │ │ │
│ │ │ │ │ │ │
│ │ ┌────────────┴───┐ ┌───┴────────────┐ │ │ │
│ │ │ VM2 │ │ VM3 │ │ │ │
│ │ │ inference- │ │ caller-worker │ │ │ │
│ │ │ worker │ │ (TypeScript) │───────┘ outbound │ │
│ │ │ (Python) │ │ │ via NAT │ │
│ │ │ No Public IP │ │ No Public IP │ │ │
│ │ └────────────────┘ └─────────────────┘ │ │
│ └──────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────┘
Client (HTTPS POST)
│
▼
Nginx :443 ──── TLS termination + rate limit (10 req/s, burst 20)
│
▼
iii-engine :3111 ──── HTTP trigger routing
│
▼ (WebSocket RPC down to VM3)
caller-worker (TypeScript, VM3) ──── request validation
│
▼ (nested RPC call back through engine)
inference-worker (Python, VM2) ──── SLM token generation
│
▼ (response bubbles back up the chain)
Client ◄──── JSON response {"choices": [...]}
| Layer | Technology | Role |
|---|---|---|
| Cloud | AWS EC2, VPC, IAM, SG | Infrastructure hosting |
| IaC | Terraform ≥ 1.5 | Reproducible provisioning |
| Config Mgmt | Ansible ≥ 2.15 | Software configuration & deployment |
| API Gateway | Nginx | TLS termination, rate limiting, reverse proxy |
| RPC Engine | iii-engine | WebSocket-based RPC coordination |
| Worker (Inference) | Python 3.11 | LLM inference via iii SDK |
| Worker (Caller) | TypeScript / Node.js | API trigger handler |
| Service Mgmt | systemd | Process supervision & auto-restart |
| OS | Ubuntu 22.04 LTS | All VMs |
| Networking | WebSockets (WS) | Worker-to-engine communication |
| NAT | Self-managed t3.nano | Private subnet outbound routing |
devops/
│
├── terraform/ # Infrastructure as Code
│ ├── main.tf # Provider configuration (AWS + dual-account)
│ ├── vpc.tf # VPC, subnets, IGW, route tables
│ ├── instances.tf # EC2 instances (Engine, Caller, Inference)
│ ├── nat_instance.tf # NAT instance + iptables bootstrap + AMI data
│ ├── security_groups.tf # Per-VM security group rules
│ ├── iam.tf # IAM roles, policies, instance profiles
│ ├── key_pair.tf # RSA 4096 key generation + local PEM file
│ ├── outputs.tf # Public IPs, private IPs, SSH commands
│ └── variables.tf # Configurable variables with sane defaults
│
├── ansible/ # Configuration Management
│ ├── playbook.yml # Master playbook — 4 plays, 5 roles
│ ├── generate_inventory.py # Reads terraform output → builds inventory.ini
│ ├── ansible.cfg # SSH settings, host key checking disabled
│ ├── inventory.ini # Auto-generated by generate_inventory.py
│ └── roles/
│ ├── common/ # System packages, swap, user creation
│ ├── nginx/ # Nginx install, SSL cert gen, config deploy
│ ├── engine/ # iii-engine install, systemd service
│ ├── inference-worker/ # Python venv, iii SDK, systemd service
│ └── caller-worker/ # Node.js, npm install, systemd service
│
├── nginx/
│ └── iii-api.conf # Nginx vhost — TLS, rate limit, proxy rules
│
├── systemd/
│ ├── iii-engine.service # systemd unit for iii-engine
│ ├── inference-worker.service # systemd unit for Python inference worker
│ └── caller-worker.service # systemd unit for TypeScript caller worker
│
├── quickstart/ # Worker source code
│ ├── config.yaml # iii-engine configuration (HTTP port, queue, state)
│ ├── workers/
│ │ ├── inference-worker/
│ │ │ └── inference_worker.py # Python worker: registers inference::run_inference
│ │ └── caller-worker/
│ │ └── src/worker.ts # TypeScript worker: HTTP trigger + RPC chain
│ └── README.md
│
├── scripts/
│ ├── deploy.sh # One-command Bash deployment pipeline
│ ├── deploy.ps1 # One-command PowerShell deployment (WSL-aware)
│ ├── teardown.sh # Terraform destroy + cleanup
│ ├── teardown.ps1 # PowerShell teardown
│ ├── test-api.sh # API smoke test script
│ ├── test-api.ps1 # PowerShell API smoke test
│ ├── bootstrap-backend.sh # S3 remote state backend setup
│ ├── config_audit_agent.py # Configuration drift detection tool
│ ├── check_sdk.sh # Target file search debugging assistant
│ ├── check_services.sh # Target services remote status analyzer
│ └── test_curl.sh # Fast diagnostic raw completions client
│
├── screenshots/ # Deployment evidence screenshots
│ └── README.md # Screenshot capture guide
│
├── diagrams/ # Architecture diagrams
│ └── architecture_diagram.png # 🎨 Master high-fidelity system topology
│
├── docs/ # 📚 Organized Documentation Center
│ ├── README.md # Documentation central index
│ ├── ARCHITECTURE.md # Deep-dive system architecture
│ ├── DEPLOYMENT.md # Step-by-step deployment guide
│ ├── SECURITY.md # Multi-layer secure parameters audit
│ ├── API_REFERENCE.md # Completions & Health endpoints specs
│ ├── TROUBLESHOOTING.md # Real incident debugging logs (12 cases)
│ ├── WRITEUP.md # 100x scalability & production hardening recommendations
│ ├── FINAL_REPORT.md # Complete implementation deliverables report
│ ├── WHY.md # Overarching system motive & choices writeup
│ ├── INFRA.md # Cloud networking HCL blueprints details
│ ├── PROJECT_ANALYSIS.md # Configuration mappings & system checks deep-dive
│ ├── CONFIG_AUDIT_REPORT.md # Dynamic security compliance audit (100/100 score)
│ ├── ALL.md # Consolidated master architectural reference
│ └── TASK.md # Granular implementation checklist log
│
├── README.md # ← You are here
Selected from shared topics, language and repository description—not editorial ratings.
KushagraB424 /
A production-grade, highly-scalable visual node-based AI workflow orchestrator.