vLLM
Open SourceInference & Routing
High-throughput inference server for production.
What it’s for
vLLM is the go-to engine for serving open models at scale. PagedAttention and continuous batching deliver many times the throughput of naive serving, behind a drop-in OpenAI-compatible API. Used across the industry for self-hosted inference.
How to use it
pip install vllm, then run vllm serve <model-name>. You get an OpenAI-compatible endpoint you can put behind a load balancer like any other service. Requires a CUDA-capable GPU.
Pricing
Free and open source (Apache 2.0). Your cost is the hardware you run it on.