BACK TO ALL DISPATCHES
AI Infrastructure8 min read
August 28, 2026

High-Throughput Generative AI: Orchestrating LLM Serving on Kubernetes with GPU Slicing

A comprehensive engineering guide to hosting and auto-scaling open-source LLM inference endpoints with vLLM, NVIDIA Multi-Instance GPU (MIG) partitioning, and custom Prometheus queue telemetry.

A
AkhilDevOps & Cloud Infrastructure
#Kubernetes#GenAI#GPU#NVIDIA#vLLM#Prometheus#Python

The GPU Utilization Dilemma in Enterprise AI

Self-hosting Large Language Models (LLMs) like Llama 3, Mistral, and DeepSeek in production presents a significant operational challenge: GPU resources (such as NVIDIA A100s or H100s) are costly, while token generation workloads fluctuate drastically throughout the day.

Running dedicated full-card GPU allocations for small or asynchronous inference requests leads to severe underutilization (often 15–25% compute utilization) and massive cloud bills.

In this deep dive, we detail the multi-tenant GPU orchestration architecture we built using **NVIDIA Multi-Instance GPU (MIG)**, **vLLM Continuous PagedAttention**, and **Kubernetes Custom Metrics HPA**.


Multi-Instance GPU (MIG) Slicing Topology

By leveraging NVIDIA Multi-Instance GPU technology on 80GB A100 nodes, we partition each physical accelerator into isolated hardware instances with dedicated high-bandwidth memory (HBM), compute SMs, and memory crossbars:

┌────────────────────────────────────────────────────────┐
│           PHYSICAL NVIDIA A100 (80GB VRAM)             │
├────────────────────┬──────────────────┬────────────────┤
│ MIG Profile: 3g.40gb│ MIG Profile: 2g.20gb│ MIG Profile: 1g.10gb│
│ (Heavy 70B Model)  │ (14B Agent Flow) │ (Embedding Svc)│
│ Dedicated 40GB HBM │ Dedicated 20GB HBM│ Dedicated 10GB HBM│
└────────────────────┴──────────────────┴────────────────┘

Each MIG instance appears as an independent, deterministic GPU device to the Kubernetes `k8s-device-plugin`, providing complete hardware memory isolation so that an Out-Of-Memory (OOM) error in one model never cascades to affect adjacent workloads.


vLLM Engine Configuration

We deploy inference pods using **vLLM**, which utilizes **PagedAttention** to eliminate KV-cache memory fragmentation and enables dynamic request batching:

yamlIMMUTABLE CONFIG
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama-serving
  namespace: ai-workloads
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: vllm-engine
        image: vllm/vllm-openai:latest
        args:
        - "--model=/models/llama-3-8b-instruct"
        - "--tensor-parallel-size=1"
        - "--gpu-memory-utilization=0.92"
        - "--max-model-len=8192"
        - "--enforce-eager"
        - "--disable-log-requests"
        resources:
          limits:
            nvidia.com/mig-3g.40gb: 1
            cpu: "8"
            memory: "32Gi"
          requests:
            nvidia.com/mig-3g.40gb: 1
            cpu: "4"
            memory: "16Gi"

Queue-Based Autoscaling with Prometheus KEDA

Standard CPU/Memory metrics are useless for autoscaling Generative AI workloads because GPU memory is allocated statically up front by the model weights and KV-cache.

Instead, we expose real-time vLLM engine metrics directly to Prometheus:

  • `vllm:num_requests_waiting`: The number of requests pending in the iteration queue.
  • `vllm:avg_generation_throughput_tok_per_s`: Instantaneous generation speed.
  • `vllm:time_to_first_token_seconds`: P95 latency before prompt prefill returns.
  • We configure **KEDA (Kubernetes Event-driven Autoscaling)** to scale our pod replicas dynamically whenever `num_requests_waiting > 4` over a 45-second sliding window, ensuring prompt responses remain sub-second even during traffic bursts.

    ABOUT THE AUTHOR

    Akhil — DevOps Engineer

    Specializing in AWS cloud infrastructure, Kubernetes container meshes, zero-downtime CI/CD automation, and Generative AI model inference clusters.