The GPU Utilization Dilemma in Enterprise AI
Self-hosting Large Language Models (LLMs) like Llama 3, Mistral, and DeepSeek in production presents a significant operational challenge: GPU resources (such as NVIDIA A100s or H100s) are costly, while token generation workloads fluctuate drastically throughout the day.
Running dedicated full-card GPU allocations for small or asynchronous inference requests leads to severe underutilization (often 15–25% compute utilization) and massive cloud bills.
In this deep dive, we detail the multi-tenant GPU orchestration architecture we built using **NVIDIA Multi-Instance GPU (MIG)**, **vLLM Continuous PagedAttention**, and **Kubernetes Custom Metrics HPA**.
Multi-Instance GPU (MIG) Slicing Topology
By leveraging NVIDIA Multi-Instance GPU technology on 80GB A100 nodes, we partition each physical accelerator into isolated hardware instances with dedicated high-bandwidth memory (HBM), compute SMs, and memory crossbars:
┌────────────────────────────────────────────────────────┐
│ PHYSICAL NVIDIA A100 (80GB VRAM) │
├────────────────────┬──────────────────┬────────────────┤
│ MIG Profile: 3g.40gb│ MIG Profile: 2g.20gb│ MIG Profile: 1g.10gb│
│ (Heavy 70B Model) │ (14B Agent Flow) │ (Embedding Svc)│
│ Dedicated 40GB HBM │ Dedicated 20GB HBM│ Dedicated 10GB HBM│
└────────────────────┴──────────────────┴────────────────┘Each MIG instance appears as an independent, deterministic GPU device to the Kubernetes `k8s-device-plugin`, providing complete hardware memory isolation so that an Out-Of-Memory (OOM) error in one model never cascades to affect adjacent workloads.
vLLM Engine Configuration
We deploy inference pods using **vLLM**, which utilizes **PagedAttention** to eliminate KV-cache memory fragmentation and enables dynamic request batching:
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-llama-serving
namespace: ai-workloads
spec:
replicas: 3
template:
spec:
containers:
- name: vllm-engine
image: vllm/vllm-openai:latest
args:
- "--model=/models/llama-3-8b-instruct"
- "--tensor-parallel-size=1"
- "--gpu-memory-utilization=0.92"
- "--max-model-len=8192"
- "--enforce-eager"
- "--disable-log-requests"
resources:
limits:
nvidia.com/mig-3g.40gb: 1
cpu: "8"
memory: "32Gi"
requests:
nvidia.com/mig-3g.40gb: 1
cpu: "4"
memory: "16Gi"Queue-Based Autoscaling with Prometheus KEDA
Standard CPU/Memory metrics are useless for autoscaling Generative AI workloads because GPU memory is allocated statically up front by the model weights and KV-cache.
Instead, we expose real-time vLLM engine metrics directly to Prometheus:
We configure **KEDA (Kubernetes Event-driven Autoscaling)** to scale our pod replicas dynamically whenever `num_requests_waiting > 4` over a 45-second sliding window, ensuring prompt responses remain sub-second even during traffic bursts.