LLM inference engine written from scratch in PyTorch, serving Qwen3-4B with paged attention and continuous batching — 12% higher output throughput than vLLM in eager mode.
PythonPyTorchCUDAInferenceFastAPI
Overview
Helios is an inference engine for Qwen3-4B on NVIDIA GPUs, written directly in PyTorch: the model, weight loading, tokenization, KV cache, scheduler, and HTTP server. Paged attention runs over a shared pool of 256-token KV blocks with per-request page tables and cross-request prefix caching. A continuous batching scheduler packs decode tokens and chunked prefill into one forward pass, and memory-aware admission control keeps active KV and cached prefixes inside a shared GPU budget. A FastAPI server exposes an OpenAI-compatible chat completions API.
Highlights
- 12% higher output throughput than vLLM in eager mode
- Native Qwen3-4B implementation, including GQA, RoPE, RMSNorm, and SwiGLU
- vLLM-inspired paged attention over a shared pool of 256-token KV-cache blocks with per-request page tables
- Cross-request prefix caching of completed prompt blocks
- Continuous batching that combines decode and chunked prefill in a single forward pass
- Memory-aware admission control with a shared budget for active KV and cached prefixes
Stack
- Python 3.11+
- Runtime, engine, and server
- PyTorch
- Qwen3 decoder, paged KV cache, and scheduler
- FastAPI
- OpenAI-compatible chat completions API
- Hugging Face
- Tokenizer, safetensors, and model snapshot loading
- CUDA
- NVIDIA GPU execution