vLLM is a free, open-source high-throughput inference engine for large language models built on PagedAttention. See features, use cases, and how it works.
vLLM is an open-source inference and serving engine for large language models designed for high throughput and efficient memory use. It was created by Woosuk Kwon during his PhD at UC Berkeley's Sky Computing Lab, with co-founders Simon Mo, Kaichao You, and Roger Wang, and academic advisors Joseph Gonzalez and Ion Stoica, and was introduced publicly in 2023.
The project centers on PagedAttention, an algorithm inspired by virtual memory paging in operating systems that manages the LLM's key-value cache in non-contiguous memory blocks, reducing waste and letting more requests be served concurrently on the same hardware.
vLLM supports a very wide range of open-source language models with an OpenAI-compatible API, so it can often act as a drop-in replacement for hosted LLM APIs. It includes continuous batching, distributed inference across multiple GPUs and nodes, and support for several quantization formats to reduce memory and compute cost.
The project reports up to 24 times higher throughput than the standard Hugging Face Transformers library in benchmark comparisons, without requiring changes to model architecture, and is now part of the PyTorch Foundation with contributions from thousands of developers and backing from major cloud and hardware vendors.
vLLM is completely free and open source. There is no vendor, license fee, or subscription; teams run it on their own infrastructure or through cloud providers that offer vLLM-based managed inference, and costs are limited to the underlying compute used to run it.
The project accepts donations through GitHub Sponsors and OpenCollective to support ongoing development, but this is optional and separate from using the software itself.
vLLM is used to self-host and serve large language models with high throughput and efficient GPU memory use, typically behind an OpenAI-compatible API.
Yes, vLLM is completely free and open source; users only pay for the compute infrastructure they run it on.
vLLM was created by Woosuk Kwon during his PhD at UC Berkeley's Sky Computing Lab, along with co-founders Simon Mo, Kaichao You, and Roger Wang.
PagedAttention is the memory management algorithm at the core of vLLM that stores the key-value cache in non-contiguous blocks, similar to virtual memory paging, to reduce waste and boost throughput.
Yes, vLLM can serve models behind an API compatible with the OpenAI Chat Completions format.
It supports a wide range of open-source LLMs, including Llama, Mistral, Qwen, and many other model families.
vLLM is maintained by an active open-source community as part of the PyTorch Foundation, with support from companies including AWS, Google Cloud, NVIDIA, AMD, and Intel.
The vLLM team reports up to 24 times higher throughput compared to the standard Hugging Face Transformers library in their benchmarks.