Stand up gpt-oss-120b
on RunPod.

Copy, paste, run. One block at a time.

Open two terminals on the same RunPod GPU Pod. Paste each whole block into Bash, including the python3 and PY lines. No notebook, OpenAI API key or Python SDK is needed.

1. 1. Check the Pod

   Run inside the GPU Pod, not on your laptop. Stop if this check fails. No packages or weights are downloaded here.

   Terminal A · Bash + Python 3 Copy block

   python3 - <<'PY'
   import shutil
   import subprocess
   from importlib.metadata import version
   
   if version("vllm") != "0.10.1":
       raise SystemExit("Expected the vllm/vllm-openai:v0.10.1 image.")
   rows = subprocess.check_output([
       "nvidia-smi", "--query-gpu=name,memory.total,memory.free",
       "--format=csv,noheader,nounits",
   ], text=True).strip().splitlines()
   if len(rows) != 1:
       raise SystemExit("This walkthrough expects exactly one A100 80GB.")
   name, total, free = [part.strip() for part in rows[0].split(",")]
   if "A100" not in name or float(total) < 79000 or float(free) < 74000:
       raise SystemExit("Use one idle A100 80GB; other GPUs are outside this tested configuration.")
   disk = shutil.disk_usage("/workspace")
   if disk.free < 100_000_000_000:
       raise SystemExit("Need at least 100GB free on /workspace; allocate the guide's 200GB model volume.")
   print(f"GPU: {name}. Free VRAM: {float(free) / 1024:.1f} GiB.")
   print(f"Free model-volume space: {disk.free / 1e9:.1f} GB.")
   print("Preflight passed. Confirm your budget, then run block 2.")
   PY

   Expected: Preflight passed. A memory check is not proof that inference will work.

2. 2. Start the model

   Downloads the checkpoint and runs vLLM in the foreground. Do not run this twice or alongside an auto-started server. Leave it running and open a second terminal on the same Pod.

   Terminal A · keep this terminal open Copy block

   python3 - <<'PY'
   import os
   import socket
   from importlib.metadata import version
   
   if version("vllm") != "0.10.1":
       raise SystemExit("Use vllm/vllm-openai:v0.10.1 for this walkthrough.")
   with socket.socket() as probe:
       if probe.connect_ex(("127.0.0.1", 8000)) == 0:
           raise SystemExit("Port 8000 is already in use. Do not launch a second server.")
   os.environ["HF_HOME"] = "/workspace/huggingface"
   command = ['vllm',
    'serve',
    'openai/gpt-oss-120b',
    '--max-model-len',
    '32768',
    '--served-model-name',
    'gpt-oss-120b',
    '--ignore-patterns',
    'original/*',
    '--host',
    '127.0.0.1',
    '--port',
    '8000']
   os.execvp(command[0], command)
   PY

   Expected: Server logs appear. The first download and load can take several minutes.

3. 3. Wait until the model is ready

   This polls for up to 15 minutes. It does not stop the Pod or cap billing. Use a shorter manual deadline if your budget requires it.

   Terminal B · same Pod Copy block

   python3 - <<'PY'
   import json
   import os
   import urllib.error
   import urllib.request
   
   # Stay inside this Pod; do not send local requests through an HTTP proxy.
   http = urllib.request.build_opener(urllib.request.ProxyHandler({}))
   headers = {"Content-Type": "application/json"}
   token = os.environ.get("VLLM_API_KEY")
   if token:
       headers["Authorization"] = "Bearer " + token
   
   import time
   
   deadline = time.monotonic() + 900
   while time.monotonic() < deadline:
       try:
           request = urllib.request.Request(
               "http://127.0.0.1:8000/v1/models", headers=headers
           )
           with http.open(request, timeout=5) as response:
               models = json.load(response).get("data", [])
           if any(item.get("id") == "gpt-oss-120b" for item in models):
               print("Ready: gpt-oss-120b. Run block 4.")
               break
           raise SystemExit("An unexpected model is serving on port 8000.")
       except urllib.error.HTTPError as error:
           if error.code in (401, 403):
               raise SystemExit("Authentication failed. Use the Pod's VLLM_API_KEY in both terminals.") from None
       except (urllib.error.URLError, TimeoutError):
           pass
       print("Waiting for model download / load. Check terminal A for errors.", flush=True)
       time.sleep(10)
   else:
       raise SystemExit("Timed out. Stop the server in terminal A and terminate the test Pod to end billing.")
   PY

   Expected: Ready: gpt-oss-120b. Run block 4.

4. 4. Run the inference check

   Checks a real final answer, not just HTTP 200. The 256-token budget gives the reasoning model room to finish.

   Terminal B · Python request Copy block

   python3 - <<'PY'
   import json
   import os
   import urllib.error
   import urllib.request
   
   # Stay inside this Pod; do not send local requests through an HTTP proxy.
   http = urllib.request.build_opener(urllib.request.ProxyHandler({}))
   headers = {"Content-Type": "application/json"}
   token = os.environ.get("VLLM_API_KEY")
   if token:
       headers["Authorization"] = "Bearer " + token
   
   prompt = 'Reply with: deployment healthy'
   payload = {
       "model": "gpt-oss-120b",
       "messages": [{"role": "user", "content": prompt}],
       "max_tokens": 256,
   }
   request = urllib.request.Request(
       "http://127.0.0.1:8000/v1/chat/completions",
       data=json.dumps(payload).encode(), headers=headers, method="POST",
   )
   try:
       with http.open(request, timeout=120) as response:
           result = json.load(response)
   except urllib.error.HTTPError as error:
       raise SystemExit(f"Inference failed: HTTP {error.code}. Check terminal A.") from None
   except (urllib.error.URLError, TimeoutError):
       raise SystemExit("Request failed or timed out. Check terminal A; the Pod still bills.") from None
   choices = result.get("choices") or []
   if not choices:
       raise SystemExit("No completion returned.")
   choice = choices[0]
   answer = (choice.get("message") or {}).get("content") or ""
   if choice.get("finish_reason") != "stop" or not answer.strip():
       raise SystemExit("No complete final answer. Inspect the server and token budget; this is not a pass.")
   if "deployment healthy" not in answer.lower():
       raise SystemExit("Unexpected answer: smoke test failed.")
   print(answer)
   print("Completion tokens:", result.get("usage", {}).get("completion_tokens", "not reported"))
   PY

   Expected: deployment healthy, followed by the completion-token count.

5. 5. Try your own prompt (optional)

   Change the prompt string and run again. Keep prompts short for this smoke test; longer outputs may need a larger token budget and more time.

   Terminal B · edit the prompt line Copy block

   python3 - <<'PY'
   import json
   import os
   import urllib.error
   import urllib.request
   
   # Stay inside this Pod; do not send local requests through an HTTP proxy.
   http = urllib.request.build_opener(urllib.request.ProxyHandler({}))
   headers = {"Content-Type": "application/json"}
   token = os.environ.get("VLLM_API_KEY")
   if token:
       headers["Authorization"] = "Bearer " + token
   
   prompt = 'In two sentences, explain what a GPU does during LLM inference.'
   payload = {
       "model": "gpt-oss-120b",
       "messages": [{"role": "user", "content": prompt}],
       "max_tokens": 512,
   }
   request = urllib.request.Request(
       "http://127.0.0.1:8000/v1/chat/completions",
       data=json.dumps(payload).encode(), headers=headers, method="POST",
   )
   try:
       with http.open(request, timeout=120) as response:
           result = json.load(response)
   except urllib.error.HTTPError as error:
       raise SystemExit(f"Inference failed: HTTP {error.code}. Check terminal A.") from None
   except (urllib.error.URLError, TimeoutError):
       raise SystemExit("Request failed or timed out. Check terminal A; the Pod still bills.") from None
   choices = result.get("choices") or []
   if not choices:
       raise SystemExit("No completion returned.")
   choice = choices[0]
   answer = (choice.get("message") or {}).get("content") or ""
   if choice.get("finish_reason") != "stop" or not answer.strip():
       raise SystemExit("No complete final answer. Inspect the server and token budget; this is not a pass.")
   print(answer)
   print("Completion tokens:", result.get("usage", {}).get("completion_tokens", "not reported"))
   PY

   Expected: A nonempty final answer. This is not a model-quality benchmark.

Reference: RunPod connections · Pod lifecycle and storage · OpenAI vLLM guide.