High-throughput open-source engine for serving large language models
Do you work on vLLM?
This listing is unclaimed. Claim it for free to edit the details, add screenshots and reply to reviews.
vLLM is an inference and serving engine for teams that host large language models themselves. It is licensed under Apache 2.0 and focuses on throughput and memory efficiency. Techniques include PagedAttention, continuous batching, prefix caching and quantization options. It exposes an OpenAI-compatible API, supports structured outputs and metrics, and comes with Docker and Kubernetes deployment guidance and integrations with agent frameworks.
From people who have used it. One review per person, checked by a human.
No reviews yet. Be the first to say something useful.