Helios

LLM inference engine written from scratch in PyTorch, serving Qwen3-4B with paged attention and continuous batching — 12% higher output throughput than vLLM in eager mode.

PythonPyTorchCUDAInferenceFastAPI
Helios

Overview

Helios is an inference engine for Qwen3-4B on NVIDIA GPUs, written directly in PyTorch: the model, weight loading, tokenization, KV cache, scheduler, and HTTP server. Paged attention runs over a shared pool of 256-token KV blocks with per-request page tables and cross-request prefix caching. A continuous batching scheduler packs decode tokens and chunked prefill into one forward pass, and memory-aware admission control keeps active KV and cached prefixes inside a shared GPU budget. A FastAPI server exposes an OpenAI-compatible chat completions API.

Highlights

  • 12% higher output throughput than vLLM in eager mode
  • Native Qwen3-4B implementation, including GQA, RoPE, RMSNorm, and SwiGLU
  • vLLM-inspired paged attention over a shared pool of 256-token KV-cache blocks with per-request page tables
  • Cross-request prefix caching of completed prompt blocks
  • Continuous batching that combines decode and chunked prefill in a single forward pass
  • Memory-aware admission control with a shared budget for active KV and cached prefixes

Stack

Python 3.11+
Runtime, engine, and server
PyTorch
Qwen3 decoder, paged KV cache, and scheduler
FastAPI
OpenAI-compatible chat completions API
Hugging Face
Tokenizer, safetensors, and model snapshot loading
CUDA
NVIDIA GPU execution