SGLang: The Complete Guide to High-Performance LLM Inference

Running large language models in production is expensive. Every millisecond of latency costs money. Every inefficient GPU cycle burns through your compute budget. And as your traffic scales, these costs compound fast. SGLang changes that equation entirely.

SGLang is an open-source, high-performance serving framework for large language models (LLMs) and multimodal models. Developed by UC Berkeley and hosted by LMSYS, it uses RadixAttention for automatic KV cache reuse, achieving up to 6x higher throughput than alternatives. SGLang powers over 400,000 GPUs in production at companies including xAI, NVIDIA, AMD, and LinkedIn.

This guide takes you from zero to production-ready SGLang deployment. You'll learn how to install SGLang using pip, Docker, or Kubernetes. You'll understand the architecture that makes it fast—particularly RadixAttention and the zero-overhead scheduler. You'll see benchmark data comparing SGLang to vLLM, and you'll get production deployment configurations that work. By the end, you'll have everything you need to deploy SGLang at scale.

What is SGLang?

SGLang (short for Structured Generation Language) is more than just another inference server. It's a complete system for efficient LLM execution, combining a Python-embedded frontend language with a highly optimized backend runtime.

The frontend provides primitives for defining complex generation programs—things like parallel prompt execution, constrained generation, and multi-step reasoning chains. The backend handles the actual inference with innovations like RadixAttention for automatic KV cache sharing across requests.

In March 2025, SGLang was integrated into the PyTorch ecosystem, cementing its position as a production-grade tool with community support and ongoing development. The current version is v0.5.8, released January 2026.

What makes SGLang different from other inference servers? Three things:

1. Automatic optimization: RadixAttention discovers and exploits caching opportunities in your workload without configuration. Other systems require you to predict and configure caching patterns manually.
2. Full-stack approach: SGLang isn't just a backend runtime—it includes a frontend language for expressing complex generation programs. Multi-turn conversations, branching reasoning, and parallel generation all have first-class support.
3. Production-first design: From the health endpoints to the graceful degradation under load, SGLang is built for production environments with thousands of concurrent requests.

Key Features at a Glance

Feature | Description

RadixAttention | Automatic KV cache reuse across requests with shared prefixes

OpenAI-Compatible API | Drop-in replacement for existing OpenAI integrations

Multi-Modal Support | Process images, video, and text with vision-language models

Quantization | FP8, INT4, AWQ, and GPTQ for reduced memory and faster inference

Multi-GPU Scaling | Tensor, pipeline, expert, and data parallelism out of the box

Structured Output | JSON schema enforcement and constrained generation

Who Uses SGLang?

The adoption numbers speak for themselves. SGLang is deployed on over 400,000 GPUs worldwide, generating trillions of tokens daily. Major technology companies have built their inference infrastructure on SGLang:

• xAI uses SGLang for Grok inference
• AMD and NVIDIA include SGLang in their AI software stacks
• LinkedIn, Cursor, and other tech companies run production workloads
• Cloud providers including Oracle Cloud, Google Cloud, Microsoft Azure, and AWS offer SGLang-compatible deployments

The SGLang GitHub repository is the central hub for the project, with active development and community contributions. The project has grown rapidly—from research prototype to production infrastructure powering some of the world's largest AI deployments.

Why Choose SGLang? Performance Benefits

Speed matters in LLM inference. Every millisecond of latency affects user experience. Every token per second of throughput determines how many requests you can serve per GPU. SGLang delivers on both fronts through three key innovations.

RadixAttention for Automatic KV Cache Reuse

The biggest performance win comes from RadixAttention, SGLang's approach to KV cache management.

Here's the problem it solves: In typical inference scenarios, many requests share common prefixes. Chat applications have system prompts. Few-shot learning uses the same examples across requests. Multi-turn conversations share history. Without cache reuse, every request recomputes the KV cache for these shared prefixes—wasting both GPU memory and computation.

RadixAttention fixes this by storing KV cache tensors in a radix tree data structure. When a new request arrives, SGLang automatically matches its prefix against cached entries and reuses what it can. This happens without any configuration—you don't need to manually define caching strategies or predict which prefixes will be shared.

The result: reduced memory usage, larger batch sizes, and dramatically faster time-to-first-token for requests that hit the cache.

Zero-Overhead CPU Scheduler

While RadixAttention optimizes memory, the scheduler optimizes GPU utilization.

Traditional inference servers have a serial pattern: prepare batch on CPU, execute on GPU, prepare next batch, execute, and so on. The GPU sits idle during batch preparation, which can take milliseconds—time that adds up across millions of requests.

SGLang eliminates this overhead with an overlapped scheduling architecture. While the GPU processes the current batch, the CPU prepares the next one. When the GPU finishes, the next batch is already waiting. The result: GPU utilization approaches 100% on sustained workloads.

The scheduler also implements intelligent request ordering to maximize cache hits in the radix tree. Requests with common prefixes get batched together when possible, increasing the cache hit rate and reducing redundant computation.

Continuous batching further improves efficiency. Instead of waiting for an entire batch to complete before starting the next one, SGLang inserts new requests into the batch as slots become available. A request that finishes generating doesn't leave its slot empty—a new request immediately takes its place.

Performance Numbers That Matter

Benchmarks on H100 GPUs running Llama 3.1 8B with ShareGPT workloads show SGLang's advantages:

flowchart TB
    subgraph Clients["Client Requests"]
        R1["Request 1"]
        R2["Request 2"]
        R3["Request 3"]
    end

    subgraph Frontend["Frontend API Server"]
        API["OpenAI-Compatible API"]
        TOK["Tokenizer"]
    end

    subgraph Scheduler["Zero-Overhead CPU Scheduler"]
        direction TB
        QUEUE["Request Queue"]
        BATCH["Batch Assembly"]
        PREP["Prepare Next Batch<br/>(while GPU busy)"]
    end

    subgraph RadixCache["RadixAttention KV Cache"]
        direction TB
        ROOT["Root Node"]
        SYS["System Prompt<br/>[cached]"]
        USR1["User Context A"]
        USR2["User Context B"]
        KV1["KV Tensors"]
        KV2["KV Tensors"]

        ROOT --> SYS
        SYS --> USR1
        SYS --> USR2
        USR1 --> KV1
        USR2 --> KV2
    end

    subgraph GPU["GPU Execution"]
        direction TB
        PREFILL["Prefill<br/>(process prompts)"]
        DECODE["Decode<br/>(generate tokens)"]
        CBATCH["Continuous Batching<br/>(insert new requests)"]
    end

    subgraph Output["Response Stream"]
        RESP["Streaming Tokens"]
    end

    R1 --> API
    R2 --> API
    R3 --> API
    API --> TOK
    TOK --> QUEUE
    QUEUE --> BATCH
    BATCH --> PREP

    PREP -->|"Check prefix matches"| RadixCache
    RadixCache -->|"Reuse cached KV"| PREFILL
    PREP -->|"Submit batch"| PREFILL

    PREFILL --> DECODE
    DECODE --> CBATCH
    CBATCH -->|"Slot available"| QUEUE
    DECODE --> RESP

    style RadixCache fill:#e6f3ff,stroke:#0066cc
    style Scheduler fill:#f0f0f0,stroke:#666666
    style GPU fill:#fff0e6,stroke:#cc6600

Figure 1: SGLang Performance Architecture — The three pillars working together: RadixAttention manages KV cache reuse, the zero-overhead scheduler prepares batches while the GPU is busy, and continuous batching maximizes throughput.

• Throughput: 16,215 tokens/second vs vLLM's 12,553—a 29% advantage
• Time-to-First-Token (TTFT): 79ms mean vs 103ms—23% faster
• Inter-Token Latency (ITL): 6.0ms vs 7.1ms—15% improvement
• Concurrency Stability: SGLang maintains 30-31 tok/s under high load while vLLM degrades from 22 to 16 tok/s

These aren't theoretical numbers. They come from independent benchmarks using production-representative ShareGPT workloads on H100 GPUs.

How to Install SGLang

Getting SGLang running takes about five minutes with pip, or one command with Docker. Here are all your options.

System Requirements

Before installing, verify your system meets these requirements:

Requirement | Minimum | Recommended

Operating System | Ubuntu 20.04 | Ubuntu 22.04

Python | 3.8 | 3.10+

CUDA | 11.8 | 12.x

GPU | sm75+ (T4, A10) | H100, A100

RAM | 32 GB | 64 GB

Disk Space | 50 GB | 100 GB

Method 1: Install with pip (Recommended)

The pip installation is the fastest path to a working SGLang setup:

# Upgrade pip and install uv (faster package manager)
pip install --upgrade pip
pip install uv

# Install SGLang with all features
uv pip install "sglang[all]"

# For CUDA 12.9+, you may need to reinstall torch
uv pip install "torch==2.9.1" "torchvision" \
    --extra-index-url https://download.pytorch.org/whl/cu129 \
    --force-reinstall

If you see CUDA_HOME environment variable is not set, fix it with:

export CUDA_HOME=/usr/local/cuda

Quick install summary:

1. Verify system requirements (CUDA 11.8+, 32GB RAM, 50GB disk)
2. Install uv package manager: pip install uv
3. Install SGLang: uv pip install "sglang[all]"
4. Set CUDA_HOME if needed: export CUDA_HOME=/usr/local/cuda
5. Launch server: python -m sglang.launch_server --model-path <model>
6. Verify with test request: curl http://localhost:30000/health

Method 2: Install with Docker

Docker provides a consistent environment without dependency management:

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=$HF_TOKEN" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path meta-llama/Llama-3.1-8B-Instruct \
        --host 0.0.0.0 \
        --port 30000

Key flags explained:

• --gpus all: Expose all GPUs to the container
• --shm-size 32g: Shared memory for NCCL communication
• --ipc=host: Required for multi-process coordination
• -v ~/.cache/huggingface:/root/.cache/huggingface: Persist downloaded models

For production, use the runtime image which is 40% smaller:

lmsysorg/sglang:latest-runtime

Images are available on Docker Hub.

Method 3: Install from Source

For custom modifications or contributing to SGLang:

git clone -b v0.5.8 https://github.com/sgl-project/sglang.git
cd sglang
pip install --upgrade pip
pip install -e "python[all]"

Method 4: Kubernetes Deployment

For orchestrated deployments, SGLang supports Kubernetes with LeaderWorkerSet for multi-node inference:

kubectl apply -f docker/k8s-sglang-service.yaml

We'll cover Kubernetes deployment in detail in the Production Deployment section.

Method 5: Cloud Platform Options

SGLang is available on major cloud platforms with pre-configured environments:

Platform | Deployment Method | Notes

AWS | SageMaker, EKS, or EC2 with GPU AMI | Use Deep Learning AMI for pre-installed CUDA

Google Cloud | GKE with GPU node pools, Vertex AI | Native Kubernetes support

Azure | AKS with GPU nodes, Azure ML | Container-based deployments

Oracle Cloud | OCI with GPU shapes | SGLang partnership for optimized images

SkyPilot | Multi-cloud orchestration | Automatic cloud selection for cost optimization

SkyPilot deployment example:

# Install SkyPilot
pip install skypilot

# Deploy SGLang on the cheapest available cloud
sky launch -c sglang-cluster examples/sglang.yaml

SkyPilot automatically selects the most cost-effective cloud provider and handles instance provisioning, making it ideal for teams running across multiple clouds.

Verify Your Installation

Launch a server with a small model to verify everything works:

python -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --host 0.0.0.0 \
    --port 30000

Wait for the server to report ready, then test with a request:

curl http://localhost:30000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "meta-llama/Llama-3.1-8B-Instruct",
        "messages": [{"role": "user", "content": "Hello!"}],
        "max_tokens": 50
    }'

You should receive a JSON response with the model's completion.

SGLang Architecture and Core Concepts

Understanding how SGLang works helps you optimize it for your workloads. The architecture has three main components, with RadixAttention as the key innovation.

How RadixAttention Works

RadixAttention is SGLang's core innovation for automatic KV cache reuse. It stores KV cache tensors in a radix tree data structure, enabling efficient prefix matching and sharing across requests. Unlike vLLM's PagedAttention which uses fixed-size blocks, RadixAttention dynamically allocates memory and automatically discovers caching opportunities without manual configuration.

graph TB
    subgraph RadixTree["RadixAttention: Radix Tree KV Cache"]
        ROOT["Root"]

        SYS["System Prompt Tokens<br/>[You are a helpful assistant...]<br/>KV Cache: 2.1 MB"]

        CHAT1["Chat Session A<br/>[User: How do I...]<br/>KV Cache: 0.8 MB"]
        CHAT2["Chat Session B<br/>[User: Explain the...]<br/>KV Cache: 0.6 MB"]
        CHAT3["Chat Session C<br/>[User: What is...]<br/>KV Cache: 0.4 MB"]

        TURN1A["Turn 2A<br/>[Follow-up question]<br/>KV Cache: 0.3 MB"]
        TURN1B["Turn 2B<br/>[Different follow-up]<br/>KV Cache: 0.5 MB"]

        ROOT --> SYS
        SYS --> CHAT1
        SYS --> CHAT2
        SYS --> CHAT3
        CHAT1 --> TURN1A
        CHAT1 --> TURN1B

        LRU["LRU Eviction<br/>(least recently used)"]
        CHAT3 -.->|"Candidate for eviction"| LRU
    end

    style SYS fill:#e6f3ff,stroke:#0066cc,stroke-width:3px
    style CHAT1 fill:#e6ffe6,stroke:#00cc00
    style CHAT2 fill:#e6ffe6,stroke:#00cc00
    style CHAT3 fill:#fff0e6,stroke:#cc6600
    style LRU fill:#ffe6e6,stroke:#cc0000

Figure 2: RadixAttention Tree Structure — KV cache tensors stored in a radix tree. Shared prefixes (like system prompts) are computed once and reused across sessions. LRU eviction removes least-used leaves when memory fills.

The radix tree structure:

A radix tree (also called a compressed trie) stores token sequences as paths from root to leaf. Each edge can hold a variable-length sequence of tokens, not just single tokens. The values stored at each node are the corresponding KV cache tensors, kept in GPU memory.

When a new request arrives, SGLang walks the tree to find the longest matching prefix. If the request shares its first 1,000 tokens with a cached entry, those tokens don't need prefill computation—the KV cache is already available.

Memory management:

GPU memory is finite, so RadixAttention implements an LRU (Least Recently Used) eviction policy. When memory fills up, the least recently accessed leaf nodes are evicted first. This happens recursively—if evicting a leaf makes its parent a leaf, that parent becomes a candidate for future eviction.

The memory management system divides GPU memory into three pools:

1. Model weights: Fixed allocation for the model parameters. Size depends on model size and quantization.
2. KV cache: Controlled by --mem-fraction-static. This is where RadixAttention stores cached tensors.
3. Activation memory: Reserved for intermediate computations during forward passes.

The --mem-fraction-static parameter controls how much GPU memory goes to the KV cache. The default of 0.9 (90%) is aggressive—it works well when your workload benefits from large cache sizes. For workloads with many long sequences or high batch sizes, lowering this to 0.8 or even 0.7 provides more headroom for activations.

Why this beats PagedAttention:

vLLM's PagedAttention divides the KV cache into fixed-size blocks, like pages in virtual memory. This works well for predictable workloads where you can configure block sizes appropriately. But it requires you to predict your caching patterns.

RadixAttention takes a different approach: let the system discover patterns automatically. The radix tree naturally captures whatever prefix sharing exists in your actual traffic. No configuration needed.

Best use cases for RadixAttention:

• Chat applications with shared system prompts
• Few-shot learning where examples are reused across requests
• Multi-turn conversations with growing history
• Tree-of-thought reasoning with branching explorations

System Components

flowchart LR
    subgraph Client["Client Application"]
        APP["Your App"]
        SDK["OpenAI SDK<br/>(Python/Node/etc)"]
    end

    subgraph SGLang["SGLang Runtime"]
        subgraph Frontend["Frontend API Server"]
            HTTP["HTTP Server"]
            OAPI["OpenAI-Compatible<br/>Endpoints"]
            VALID["Request Validation"]
        end

        subgraph Tokenizer["Tokenizer Server"]
            ENC["Text → Tokens"]
            DEC["Tokens → Text"]
            VOCAB["Vocabulary Cache"]
        end

        subgraph Backend["Backend Scheduler"]
            QUEUE["Request Queue"]
            BATCH["Batch Manager"]
            RADIX["RadixAttention<br/>KV Cache"]
            EXEC["Execution Engine"]
        end
    end

    subgraph Hardware["GPU Hardware"]
        GPU1["GPU 0"]
        GPU2["GPU 1"]
        GPUN["GPU N"]
    end

    APP --> SDK
    SDK -->|"HTTP/REST"| HTTP
    HTTP --> OAPI
    OAPI --> VALID
    VALID --> ENC
    ENC --> QUEUE
    QUEUE --> BATCH
    BATCH --> RADIX
    RADIX --> EXEC
    EXEC --> GPU1
    EXEC --> GPU2
    EXEC --> GPUN
    GPU1 --> DEC
    GPU2 --> DEC
    GPUN --> DEC
    DEC -->|"Streaming Response"| HTTP

    style Frontend fill:#e6f3ff,stroke:#0066cc
    style Tokenizer fill:#f0f0f0,stroke:#666666
    style Backend fill:#fff0e6,stroke:#cc6600

Figure 3: SGLang System Components — Three main components work together: Frontend API Server handles requests, Tokenizer converts text/tokens, and Backend Scheduler orchestrates GPU execution with RadixAttention.

SGLang's runtime has three main components:

1. Frontend API Server: Handles incoming requests and implements OpenAI-compatible endpoints. Your existing code using the OpenAI Python client works without modification.
2. Tokenizer Server: Converts text to tokens and back. Runs as a separate process to avoid blocking GPU execution.
3. Backend Scheduler: The brain of the system. Manages the radix tree, decides which requests to batch together, and orchestrates GPU execution.

Supported Models and Hardware

SGLang supports over 60 model families across multiple categories:

Category | Models

Language Models | Llama, Qwen, DeepSeek, Mistral, Gemma, GLM, Kimi, MiMo, Nemotron

Embedding Models | e5-mistral, gte, mcdse

Reward Models | Skywork

Multi-Modal Models | LLaVA, Qwen-VL, InternVL, Pixtral

Diffusion Models | WAN, Qwen-Image

Hardware support spans multiple vendors:

Vendor | Supported Hardware

NVIDIA | GB200, B300, H100, A100, A10G, L4, L40S, T4 (sm75+)

AMD | MI355, MI300

Intel | Xeon CPUs, Gaudi accelerators

Google | TPUs (via SGLang-Jax backend)

Ascend | NPUs

SGLang provides day-0 support for new model releases, including recent additions like DeepSeek-V3.2 with Sparse Attention and Mistral Large 3.

Understanding the request flow:

When you send a request to SGLang, it flows through the system like this:

1. Frontend receives request: The API server parses your HTTP request and validates the input.
2. Tokenization: The tokenizer converts your text prompt into token IDs.
3. Prefix matching: The scheduler checks the radix tree for matching prefixes in the KV cache.
4. Batch assembly: Your request joins a batch of other requests for efficient processing.
5. GPU execution: The batch runs through the model, reusing cached KV values where possible.
6. Response streaming: Tokens are generated and streamed back as they're produced.

This pipeline is optimized for minimal latency at every stage. The CPU and GPU work in parallel, the scheduler maximizes cache utilization, and continuous batching keeps the GPU fully utilized.

SGLang vs vLLM: Which Should You Choose?

Both SGLang and vLLM are excellent inference engines. The right choice depends on your specific workload characteristics.

Performance Benchmarks

Head-to-head benchmarks on H100 GPUs tell a clear story:

Metric | SGLang | vLLM | Difference

Throughput | 16,215 tok/s | 12,553 tok/s | SGLang +29%

Mean TTFT | 79 ms | 103 ms | SGLang 23% faster

Inter-Token Latency | 6.0 ms | 7.1 ms | SGLang 15% faster

Cache Mechanism | RadixAttention | PagedAttention | —

Multi-turn Cache Boost | ~20% | ~15% | SGLang +5pp

High Concurrency | Stable | Degrades | SGLang wins

These benchmarks used Llama 3.1 8B with production-representative ShareGPT prompts. The 29% throughput advantage persists even when vLLM uses the FlashInfer backend, suggesting the gap comes from architectural differences rather than kernel performance.

Architectural Differences

The fundamental difference is in cache management philosophy:

SGLang (RadixAttention): Dynamic, automatic prefix discovery. The system learns what to cache from actual traffic patterns. Zero configuration required.

vLLM (PagedAttention): Fixed-size blocks with Automatic Prefix Caching (APC). Works best when you can predict and configure caching patterns.

Neither approach is universally better—they excel at different workloads.

When to Choose SGLang

flowchart TD
    START["Which inference engine<br/>should I use?"]

    Q1{"Multi-turn<br/>conversations?"}
    Q2{"Unpredictable<br/>dialog flows?"}
    Q3{"Batch inference<br/>with templates?"}
    Q4{"Need fine-grained<br/>cache control?"}
    Q5{"High concurrency<br/>with stable latency?"}

    SGLANG["Use SGLang"]
    VLLM["Use vLLM"]
    DEFAULT["SGLang<br/>(recommended default)"]

    START --> Q1

    Q1 -->|"Yes"| SGLANG
    Q1 -->|"No"| Q3

    Q3 -->|"Yes"| Q4
    Q3 -->|"No"| Q2

    Q2 -->|"Yes"| SGLANG
    Q2 -->|"No"| Q5

    Q4 -->|"Yes"| VLLM
    Q4 -->|"No"| DEFAULT

    Q5 -->|"Yes"| SGLANG
    Q5 -->|"No"| DEFAULT

    style SGLANG fill:#e6f3ff,stroke:#0066cc,stroke-width:3px
    style VLLM fill:#f0f0f0,stroke:#666666,stroke-width:3px
    style DEFAULT fill:#e6ffe6,stroke:#00cc00,stroke-width:3px

Figure 4: SGLang vs vLLM Decision Tree — Quick decision guide based on your workload characteristics. Multi-turn conversations and unpredictable dialog flows favor SGLang; batch processing with templates may favor vLLM.

Choose SGLang when:

• Multi-turn conversations dominate your traffic. Chat applications, tutoring systems, and coding assistants benefit from RadixAttention's automatic context sharing.
• Dialog flows are unpredictable. If users take conversations in different directions, SGLang's dynamic caching adapts automatically.
• You want zero configuration. RadixAttention discovers caching opportunities without tuning.
• Latency consistency matters. SGLang maintains stable performance under high concurrency where vLLM can degrade.

When to Choose vLLM

Choose vLLM when:

• Batch inference with templates is your primary use case. Document processing, data extraction, and other templated workloads with predictable prefixes.
• You need fine-grained cache control. vLLM's APC gives you explicit control over what gets cached.
• Ecosystem compatibility is critical. vLLM has broader framework integrations and a larger community.
• Your workloads are simple and predictable. Single-turn requests with consistent patterns.

Migration Considerations

Already running vLLM? Migration to SGLang is straightforward because of API compatibility:

1. No client code changes: SGLang's OpenAI-compatible API means your existing code works unchanged.
2. Same model formats: SGLang supports the same HuggingFace model formats as vLLM.
3. Gradual rollout: Run both systems in parallel and shift traffic gradually.
4. Key differences to account for:

• Server launch parameters differ (review the documentation)
• Prefix caching is automatic in SGLang (no configuration needed)
• Some advanced vLLM features may not have direct equivalents

Production Deployment Guide

Moving from development to production requires attention to containerization, orchestration, and performance tuning.

Docker Production Configuration

For production Docker deployments, use the runtime image and configure for stability:

docker run -d \
    --gpus all \
    --shm-size 32g \
    --ipc=host \
    --restart unless-stopped \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=$HF_TOKEN" \
    --name sglang-prod \
    lmsysorg/sglang:latest-runtime \
    python3 -m sglang.launch_server \
        --model-path meta-llama/Llama-3.1-8B-Instruct \
        --host 0.0.0.0 \
        --port 30000 \
        --mem-fraction-static 0.9 \
        --max-running-requests 256

Key production settings:

• -d: Run detached
• --restart unless-stopped: Auto-restart on failure
• latest-runtime: 40% smaller image without dev tools
• --mem-fraction-static 0.9: Allocate 90% of GPU memory to KV cache

Kubernetes Deployment with LeaderWorkerSet

For Kubernetes deployments, SGLang uses LeaderWorkerSet (LWS) to coordinate multi-node inference:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sglang
  labels:
    app: sglang
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sglang
  template:
    metadata:
      labels:
        app: sglang
    spec:
      containers:
      - name: sglang
        image: lmsysorg/sglang:latest-runtime
        resources:
          limits:
            nvidia.com/gpu: 1
        ports:
        - containerPort: 30000
        args:
        - python3
        - -m
        - sglang.launch_server
        - --model-path
        - meta-llama/Llama-3.1-8B-Instruct
        - --host
        - "0.0.0.0"
        - --port
        - "30000"
        livenessProbe:
          httpGet:
            path: /health
            port: 30000
          initialDelaySeconds: 120
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /health
            port: 30000
          initialDelaySeconds: 60
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: sglang
spec:
  type: ClusterIP
  ports:
  - port: 30000
    targetPort: 30000
  selector:
    app: sglang

For multi-node deployments with tensor parallelism across nodes, install LeaderWorkerSet and use the SGLang-provided manifests.

Multi-GPU Configuration

Large models require multiple GPUs. SGLang supports several parallelism strategies:

Tensor Parallelism (TP): Splits model layers across GPUs. Use --tp N where N is the number of GPUs. This is the most common approach for models that fit in combined GPU memory.

# Run Llama 70B across 8 GPUs
python -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-70B-Instruct \
    --tp 8 \
    --host 0.0.0.0 --port 30000

Data Parallelism (DP): Runs multiple model replicas for higher throughput. Use --dp N with --tp M where you have N×M GPUs total.

# 2 replicas, each using 4 GPUs (8 GPUs total)
python -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-70B-Instruct \
    --tp 4 --dp 2 \
    --host 0.0.0.0 --port 30000

Pipeline Parallelism (PP): Splits model layers sequentially across nodes. Useful when tensor parallelism isn't sufficient. Combine with TP for large-scale deployments.

# 2 nodes with 8 GPUs each, using tensor and pipeline parallelism
python -m sglang.launch_server \
    --model-path <very-large-model> \
    --tp 8 --pp 2 \
    --host 0.0.0.0 --port 30000

For expert models like Mixtral or DeepSeek MoE, SGLang also supports Expert Parallelism (EP) to distribute expert parameters across GPUs.

Performance Tuning Parameters

The key server parameters for production tuning:

Parameter | Default | Description | Tuning Advice

--tp | 1 | Tensor parallelism degree | Set to number of GPUs for large models

--mem-fraction-static | 0.9 | GPU memory for KV cache | Lower to 0.8 if hitting OOM during decode

--max-running-requests | varies | Concurrent request limit | Lower if hitting OOM, raise for throughput

--chunked-prefill-size | varies | Prefill chunk size | Lower to 4096 if prefill OOM

--enable-deterministic-inference | false | Deterministic mode | Enable if reproducibility matters

Start with defaults and adjust based on observed behavior. Monitor GPU memory usage and request latencies to guide tuning.

Monitoring and Health Checks

SGLang exposes a /health endpoint for load balancer health checks:

curl http://localhost:30000/health

The health endpoint returns HTTP 200 when the server is ready to accept requests. Use this for Kubernetes probes and load balancer configurations.

Key metrics to monitor:

Metric | What It Tells You | How to Monitor

Throughput (tok/s) | Overall system capacity | Server logs

Time to First Token | User-perceived responsiveness | Application metrics

Inter-Token Latency | Generation smoothness | Application metrics

GPU Memory Usage | Risk of OOM | nvidia-smi

Queue Depth | Request backlog | Server logs

Cache Hit Rate | RadixAttention effectiveness | Server logs (verbose mode)

Production monitoring recommendations:

1. Log aggregation: Send SGLang logs to a central system (ELK, Datadog, CloudWatch) to track throughput and latency over time.
2. GPU monitoring: Use nvidia-smi dmon for continuous GPU metrics, or tools like DCGM Exporter for Prometheus integration.
3. Application-level tracking: Instrument your client code to measure end-to-end latencies and token throughput from the user's perspective.
4. Alerting thresholds: Set alerts for:

• GPU memory above 95% (OOM risk)
• Request queue depth increasing over time
• P99 latency exceeding SLA targets
• Health check failures

Troubleshooting Common Issues

When things go wrong, these solutions address the most common problems.

Out of Memory (OOM) Errors

OOM errors have different causes requiring different fixes:

OOM during prefill (processing the input prompt):

# Reduce chunk size for prefill
python -m sglang.launch_server \
    --model-path <model> \
    --chunked-prefill-size 4096  # or 2048 for severe cases

OOM during decode (generating output tokens):

# Limit concurrent requests
python -m sglang.launch_server \
    --model-path <model> \
    --max-running-requests 128  # lower from default

General OOM (not clear which phase):

# Reduce KV cache memory allocation
python -m sglang.launch_server \
    --model-path <model> \
    --mem-fraction-static 0.8  # or 0.7 for more headroom

CUDA and Kernel Errors

CUDA errors often stem from environment issues:

# Check your environment
python3 -m sglang.check_env

# Set CUDA_HOME if missing
export CUDA_HOME=/usr/local/cuda

# For B300/GB300 GPUs specifically
export TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas

If you see kernel compilation errors, try the Triton backend:

python -m sglang.launch_server \
    --model-path <model> \
    --attention-backend triton \
    --sampling-backend pytorch

Server Hangs and Performance Issues

If the server hangs during initialization:

1. Check available GPU memory: nvidia-smi
2. Reduce --mem-fraction-static to leave more headroom
3. Reduce --cuda-graph-max-bs for CUDA graph compilation

If performance degrades over time, it may indicate memory fragmentation. A restart often resolves this.

Non-Deterministic Results

Seeing different outputs for identical requests at temperature=0? This is expected behavior due to dynamic batching.

Why it happens: Different batch sizes trigger different CUDA kernels with minor numerical differences. Dynamic batching accounts for ~95% of non-determinism, prefix caching for ~5%.

Solution:

python -m sglang.launch_server \
    --model-path <model> \
    --enable-deterministic-inference

Note: Deterministic mode has a small performance cost.

Model Loading Issues

If the server fails to load a model:

Model not found:

# Ensure HuggingFace token is set for gated models
export HF_TOKEN=hf_your_token_here

# Or pass it as a server argument
python -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --hf-token hf_your_token_here

Insufficient GPU memory:

# Use quantization to reduce memory requirements
python -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-70B-Instruct \
    --quantization fp8  # or awq, gptq

Multi-GPU not detected:

# Verify GPUs are visible
nvidia-smi -L

# Check CUDA is properly configured
python -c "import torch; print(torch.cuda.device_count())"

Troubleshooting Quick Reference

Problem | Likely Cause | Solution

OOM during prefill | Long prompts | --chunked-prefill-size 4096

OOM during decode | Too many concurrent requests | --max-running-requests 128

General OOM | KV cache too large | --mem-fraction-static 0.8

CUDA_HOME not set | Environment issue | export CUDA_HOME=/usr/local/cuda

Kernel compilation error | Backend incompatibility | --attention-backend triton

Server hangs at startup | Memory allocation | Reduce --mem-fraction-static

Non-deterministic outputs | Dynamic batching (expected) | --enable-deterministic-inference

Model not found | Missing HuggingFace token | export HF_TOKEN=your_token

Quick Reference

Essential Commands

# Install
pip install uv && uv pip install "sglang[all]"

# Launch server
python -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --host 0.0.0.0 --port 30000

# Test request
curl http://localhost:30000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model": "meta-llama/Llama-3.1-8B-Instruct",
         "messages": [{"role": "user", "content": "Hello!"}]}'

# Check environment
python3 -m sglang.check_env

# Docker (production)
docker run -d --gpus all --shm-size 32g --ipc=host \
    -p 30000:30000 lmsysorg/sglang:latest-runtime \
    python3 -m sglang.launch_server --model-path <model> \
    --host 0.0.0.0 --port 30000

Key Parameters

Parameter | Purpose

--tp N | Tensor parallelism across N GPUs

--mem-fraction-static 0.9 | GPU memory for KV cache

--max-running-requests N | Max concurrent requests

--chunked-prefill-size N | Prefill chunk size

--enable-deterministic-inference | Reproducible outputs

Useful Links

• GitHub: github.com/sgl-project/sglang
• Documentation: docs.sglang.io
• Docker Hub: hub.docker.com/r/lmsysorg/sglang
• LMSYS Blog: lmsys.org/blog/2024-01-17-sglang/

Conclusion

SGLang has established itself as the high-performance standard for LLM inference. With RadixAttention providing automatic KV cache optimization, a zero-overhead scheduler maximizing GPU utilization, and 29% higher throughput than vLLM in benchmarks, it delivers real cost savings at scale.

In this guide, you've learned:

• Installation via pip, Docker, or Kubernetes
• Architecture fundamentals including how RadixAttention works
• Performance benchmarks comparing SGLang to vLLM
• Production deployment patterns with tuning guidance
• Troubleshooting for common issues

Your next steps:

1. Try it locally: Install with pip install uv && uv pip install "sglang[all]" and run a model
2. Benchmark your workload: Use python -m sglang.bench_serving with your actual traffic patterns
3. Explore advanced features: Structured output, multi-modal models, and the SGLang frontend language

Need managed SGLang deployment? Inference.net handles the infrastructure so you can focus on your application.

Frequently Asked Questions

Is SGLang open source?

Yes. SGLang is fully open source under the Apache 2.0 license. The code is hosted on GitHub with active community development and contributions welcome.

What does SGLang stand for?

SGLang stands for "Structured Generation Language." The name reflects its dual nature: both a structured programming language for expressing complex LLM workflows and a high-performance inference runtime.

Does SGLang support streaming responses?

Yes. SGLang supports Server-Sent Events (SSE) streaming through its OpenAI-compatible API. Set "stream": true in your request to receive tokens as they're generated.

References

1. SGLang GitHub Repository - https://github.com/sgl-project/sglang
2. SGLang Official Documentation - https://docs.sglang.io/
3. LMSYS Blog: Fast and Expressive LLM Inference with RadixAttention and SGLang - https://lmsys.org/blog/2024-01-17-sglang/
4. SGLang arXiv Paper - https://arxiv.org/abs/2312.07104
5. PyTorch Blog: SGLang Joins PyTorch Ecosystem - https://pytorch.org/blog/sglang-joins-pytorch/
6. AIMultiple: LLM Inference Engines Comparison - https://research.aimultiple.com/inference-engines/
7. RunPod: SGLang vs vLLM KV Cache Analysis - https://www.runpod.io/blog/sglang-vs-vllm-kv-cache