Overview
ModelExpress is a Rust-based service that manages the complete model weight lifecycle in the cluster—from acquisition to GPU memory. It accelerates LLM inference by caching, routing, and transferring weights through the fastest available path. Deploy standalone or as a sidecar alongside vLLM, NVIDIA Dynamo, and other inference runtimes.
| LLM serving problem | How ModelExpress helps |
|---|
| Models take too long to load | GPU-to-GPU transfer via NIXL/RDMA instead of loading from storage. In P2P mode, weights already serving inference act as the cache—no extra storage. |
| JIT warmup dominates startup | Compatible vLLM TorchInductor, Triton, DeepGEMM, TileLang, CuTe DSL, and FlashInfer JIT caches transfer from a ready replica instead of being rebuilt. |
| Many nodes need the same model | Metadata backends (Redis, K8s CRD) coordinate sharing: one node loads; others receive via P2P or local paths. |
How ModelExpress manages weights in the cluster
ModelExpress orchestrates the full flow—from download to GPU memory. It ensures only one node downloads a model from external sources (e.g., HuggingFace); other nodes receive weights via P2P or shared storage—eliminating duplicate downloads and reducing cluster ingress.
- Download from HuggingFace — One node pulls the model; ModelExpress coordinates so no other node duplicates this download, reducing external ingress. In air-gapped mode, serve from cache only (
HF_HUB_OFFLINE=1).
- Persist to disk — Store in a cache backed by disk:
- Host-attached disk — Local disk on the node (single-node or per-node cache).
- PVC — RWO (ReadWriteOnce) for single-node; RWX (ReadWriteMany) for shared access across nodes.
- Disk to GPU — Inference engine (vLLM, etc.) loads weights from the cache (disk) into GPU memory.
- P2P transfer — Additional nodes receive weights via GPU-to-GPU RDMA from the first node instead of reading from disk—no duplicate downloads or disk reads.
Features
- Cold start reduction — GPU-to-GPU P2P transfer over InfiniBand instead of disk load
- HuggingFace caching — PVC-backed cache,
HF_HUB_OFFLINE, ignore_weights, get_model_path for Dynamo
- P2P GPU transfer — vLLM
modelexpress loader (mx alias) and TRT-LLM PRESHARDED loader with NVIDIA NIXL over RDMA
- JIT cache transfer — Reuse compatible vLLM compilation caches when replicas scale out
- Metadata backends — In-memory, Redis, or Kubernetes CRD (layered write-through for HA)
- Kubernetes — Helm chart, CRDs/Redis for P2P, no-shared-storage support
- CLI — Health, download, list, validate, clear; init-container support for pre-warming
- ModelStreamer integration: stream weights from object storage (AWS S3, Azure Blob, GCS) with multi-engine support
- Expanded model pull providers: NGC catalog and Google Cloud Storage in addition to Hugging Face
- GDS (GPUDirect Storage): load model weights directly from NVMe into GPU memory, bypassing the CPU/DRAM copy path
Integrations
| Runtime | Integration |
|---|
| vLLM | Native --load-format modelexpress in 0.23.0+ for P2P weight and JIT cache transfer; older versions use the ModelExpress plugin, and mx is a backward-compatible alias |
| NVIDIA Dynamo (vLLM) | get_model_path API; Dynamo model cache K8s example |
| TensorRT-LLM | LoadFormat.PRESHARDED with MxLiveCheckpointLoader for P2P weight transfer (beta) — TRT-LLM examples |
| SGLang | remote_instance + modelexpress backend with transport=nixl or transport=transfer_engine — see docs/SGLANG.md |
ModelExpress Architecture
Phase 1 — Upload once: Model Source (HuggingFace Hub, NFS) downloads to the Seed Pod (GPU), which loads and postprocesses weights, registers VRAM with NIXL, and publishes metadata to the MX Server. Phase 2 — Autoscale: New pods receive weights via NIXL GPUDirect RDMA (GPU VRAM → GPU VRAM, zero-copy) from the seed GPU, using --load-format modelexpress for inference.
┌─────────────────────────────────────────────────────────────────┐
│ ModelExpress Server │
│ Health • Model • P2P Metadata • Redis/K8s CRD backends │
└──────────────────────┬──────────────────────────────────────────┘
│
┌─────────────────┼─────────────────┐
│ metadata │ │ metadata
▼ │ ▼
┌──────────────────┐ │ ┌──────────────────┐
│ Source (vLLM) │ RDMA │ │ Target (vLLM) │
│ mx loader │════════►│ │ mx loader │
│ Load → NIXL │ NIXL │ │ Receive → FP8 │
│ Publish metadata│ │ │ Serve inference │
└──────────────────┘ │ └──────────────────┘
Source and Target exchange metadata with the server for coordination; weights transfer directly over RDMA between GPUs.
- modelexpress_server: gRPC server with configurable metadata backends (Redis, Kubernetes CRD).
- modelexpress_client: Rust CLI for cache management; Python package with inference engine loaders and
MxClient for gRPC.
- modelexpress_common: Protobuf definitions, provider trait (HuggingFace), shared configuration.
See Architecture.
Quick Start
Requirements: Rust 1.90+, protoc, Docker
git clone https://github.com/ai-dynamo/modelexpress.git
cd modelexpress
# Start a local Redis instance for metadata storage
docker run -d --name redis -p 6379:6379 redis:8-alpine
cargo build
# REDIS_URL is required; the server does not fall back to localhost:6379.
REDIS_URL=redis://localhost:6379 MX_METADATA_BACKEND=redis cargo run --bin modelexpress-server
Server listens on 0.0.0.0:8001. In another terminal:
# Download a model (shared storage)
modelexpress-cli model download meta-llama/Llama-3.3-70B-Instruct
# Verify
modelexpress-cli health
Without shared storage: use --no-shared-storage for gRPC streaming.
Air-gapped: with the model already in the local HF cache, HF_HUB_OFFLINE=1 modelexpress-cli model download <model-id> resolves it without network access.
Deployment
Kubernetes (Helm)
kubectl create secret generic hf-token-secret --from-literal=HF_TOKEN=${HF_TOKEN} -n <namespace>
helm install modelexpress ./helm --namespace modelexpress --create-namespace
Override values-production.yaml for your env. Full config: helm/README.md.
P2P GPU Transfer (vLLM)
vllm serve deepseek-ai/DeepSeek-V4-Pro \
--load-format modelexpress \
--tensor-parallel-size 8 \
--trust-remote-code
vLLM 0.23.0 recognizes the load format natively; the ModelExpress Python package must still be installed in the runtime image. The first instance loads from disk, while subsequent instances receive weights via RDMA. Set MX_ARTIFACT_TRANSFER=1 to transfer compatible JIT caches as well. P2P guide · Server setup.
ModelStreamer on Kubernetes
Load model weights directly from Azure Blob Storage, S3, or a PVC-backed local path through ModelStreamer. ModelStreamer examples · vLLM recipes.
Docker
docker compose -f docker/docker-compose.yml up --build
Configuration
Precedence: CLI → env vars (MODEL_EXPRESS_*, MX_*) → YAML → defaults.
| Variable | Default | Description |
|---|
MODEL_EXPRESS_SERVER_PORT | 8001 | gRPC port |
MODEL_EXPRESS_CACHE_DIRECTORY | ./cache | Cache root |
MX_METADATA_BACKEND | (required) | redis | kubernetes |
REDIS_URL | (required for redis) | Redis connection URL. Alternatively set MX_REDIS_HOST + MX_REDIS_PORT. No localhost fallback. |
MX_SERVER_ADDRESS | localhost:8001 | Client-side gRPC server address (P2P). Recommended. |
MODEL_EXPRESS_URL | localhost:8001 | Deprecated, pending removal in a future release. Still read by all client paths and takes precedence when both are set; keep setting it during the transition. |
cargo run --bin config_gen -- --output model-express.yaml
cargo run --bin modelexpress-server -- --config model-express.yaml --validate-config
Full reference: docs/DEPLOYMENT.md.
CLI
modelexpress-cli health
modelexpress-cli model download <model-id>
modelexpress-cli model list
modelexpress-cli model validate <model-id>
modelexpress-cli model clear <model-id>
CLI Reference
Testing
cargo test
cargo test --test integration_tests
cargo run --bin test_client -- --test-model "google-t5/t5-small"
./run_integration_tests.sh
cargo bench
Documentation
Known Issues
- NIXL_ERR_REMOTE_DISCONNECT — Source restarts invalidate rkeys. Flush Redis, redeploy.
- Large model gRPC stream — May not close automatically; use client timeout.
- GDS loader does not scale with TP — Each TP rank reads full checkpoint tensors and vLLM shards them afterward, so GDS/disk reads scale with TP degree. This can reduce or reverse expected GDS speedups versus the default mmap-based disk loader; TP-aware range reads are needed for a full fix. See GDS Reads Full Checkpoint Tensors Under TP.
Roadmap
Priorities Under Development
- DRAM and NVMe-resident shard streaming: Stream shards across workers while keeping weights in DRAM and host local high-speed NVMe.
- RL workloads: Explore fast P2P transfers to optimize RL refit phase and support for weight resharding.
- Earlier weight availability: Bring weights to prefill earlier; identify prefill workers that can act as strong source nodes.
- Multi-tier cache hierarchy: Promote and demote models across DRAM, NVMe, and PVC tiers based