GB10

2 articles tagged with "GB10"

Part 4. TANREN — an Evolved Scheduler Patched into vLLM Halves Tail Latency Under Heavy Load Series

Part 4. TANREN — an Evolved Scheduler Patched into vLLM Halves Tail Latency Under Heavy Load

The scheduler — the heart of LLM serving — evolved by TANREN and A/B tested inside a real vLLM. Across 16 real-trace configurations it beats the default FCFS 15 times (p=2.6e-4), cuts mean/p99 TTFT by 35-49% under heavy load, and does no harm when idle. Includes the honest record of going 0/6 on unseen data and six rounds of diagnosis before the win.

Read →
Part 3. TANREN — Patching the Evolved Cache Policy into a Real vLLM and Measuring It Series

Part 3. TANREN — Patching the Evolved Cache Policy into a Real vLLM and Measuring It

The cache policy that won in simulation is patched into a live vLLM and measured on real hardware. The simulated win vanished at first; after fixing where and what to measure, tail latency (p99 TTFT) came out 13-16% lower under memory pressure. The prototype's costs and the conditions where it does not help are reported as measured.

Read →