Multi-GPU
AWS runbook · model rank #22

Stand up Kimi K2.5
on AWS.

Fits one high-memory node, but needs tensor/expert parallel configuration. Production teams that need VPC controls, durable model storage, and multi-node operations.

Open AWS ↗ Read the runbook
Multi-GPU: Fits one high-memory node, but needs tensor/expert parallel configuration.
Provider walkthrough

AWS setup, in the order that matters

SageMaker HyperPod with a Hugging Face model source and vLLM/SGLang worker. Use the provider console for infrastructure and the pinned runtime block for the model server.

  1. 01

    Create the HyperPod capacity

    Provision SageMaker/HyperPod GPU capacity with at least 8x H100/A100 80GB-class accelerators, outbound access to Hugging Face, and a namespace for the inference operator. Use S3 or FSx for predictable production weight loading.

  2. 02

    Store the Hugging Face token

    Create a Kubernetes Secret referenced by the model source. Never place the token directly in the endpoint YAML.

  3. 03

    Apply an InferenceEndpointConfig

    Set modelSourceType to huggingface, modelId to moonshotai/Kimi-K2.5, prefetchEnabled to true, and the worker image/arguments from the runtime block below.

  4. 04

    Verify and productionize

    Wait for the endpoint condition to become ready, invoke /v1/chat/completions, then add replicas, autoscaling, observability, and private networking.

Runtime seedvLLM · latest
docker run --rm --gpus all --ipc=host \
  -p 8000:8000 \
  -e HUGGING_FACE_HUB_TOKEN="$HF_TOKEN" \
  -v "$PWD/.hf-cache:/root/.cache/huggingface" \
  vllm/vllm-openai:latest \
  --model moonshotai/Kimi-K2.5 \
  --tensor-parallel-size 8 \
  --max-model-len 32768 \
  --trust-remote-code \
  --served-model-name kimi-k2.5

This is a reproducible starting block. Cluster paths still need the model author's distributed recipe and the provider's orchestration layer.

Smoke testOpenAI-compatible request
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "kimi-k2.5",
    "messages": [{"role": "user", "content": "Reply with: deployment healthy"}],
    "max_tokens": 32
  }'

Run this only after logs show the model loaded. A successful HTTP response is not a latency or quality benchmark.

Before you launch

Quota, license, and storage

  • Confirm SageMaker/HyperPod GPU capacity with at least 8x H100/A100 80GB-class accelerators is actually available in the selected region.
  • Read the Modified MIT terms and accept any gated-model conditions.
  • Budget at least 850GB for weights, cache, and container layers.
  • Keep HF_TOKEN in the provider secret store—not in scripts or templates.
Before real traffic

Memory, format, and shutdown

  • Record idle/free VRAM after the model loads and after a representative prompt.
  • Validate the official chat template, reasoning parser, and tool-call parser.
  • Add authentication and TLS in front of port 8000.
  • Verify the provider's stop/delete action actually ends compute billing.
Agent-assisted setup

Hand this runbook to Claude Code or Codex

Open a terminal in the repository where you want the deployment files, start claude or codex, then paste this prompt. It asks the agent to verify sources and stop before it creates billable infrastructure.

Claude Code quickstart ↗ Codex docs ↗
Deployment prompt 
Deploy Kimi K2.5 (moonshotai/Kimi-K2.5) on AWS. Use this guide as the starting context: https://www.getflops.ai/models/kimi-k2.5/aws. Use a planning floor of 8x 80GB GPUs, 850GB storage, vllm latest, and an initial context limit of 32768 tokens. Open every linked primary source and flag any mismatch instead of guessing. Create a deployment folder containing README.md, .env.example with no secrets, a pinned start script or infrastructure manifest, and smoke-test.sh. Make the endpoint OpenAI-compatible where the runtime supports it. Run local/static validation, estimate the billable resources, and stop before provisioning paid infrastructure until I approve.
Guardrails included No secrets in files · verify primary docs · approval before spend
Research trail

The two sources to check first

Moonshot AI model card Model size, context, license, chat format, and engine notes ↗ SageMaker HyperPod model deployment docs SageMaker HyperPod with a Hugging Face model source and vLLM/SGLang worker ↗

Checked 2026-07-27. This page separates sourced facts from the planning baseline; verify current runtime support and capacity before spending.

Same model, another cloud

Compare Kimi K2.5 elsewhere

RunPod Multi-GPU 8x H100/A100 80GB-class GPUs Vast.ai Multi-GPU 8x H100/A100 80GB-class GPUs Lambda Multi-GPU 8x H100/A100 80GB-class GPUs Google Cloud Multi-GPU Vertex AI GPU deployment with at least 8x H100/A100 80GB-class accelerators Azure Multi-GPU Azure ML GPU deployment with at least 8x H100/A100 80GB-class accelerators Oracle Cloud Multi-GPU OCI GPU shape with at least 8x H100/A100 80GB-class accelerators