lmdeploy
InternLM/lmdeploy · Python · Apache-2.0
Turbocharge LLM/VLM serving with 4-bit weights and a fast gRPC API.
LMDeploy combines a TurboMind inference backend with weight quantization, continuous batching and an OpenAI-compatible server, tuned for throughput on a single consumer GPU.
Score
Setup at a glance
pip install lmdeploy Deep dive
LMDeploy is InternLM's deployment toolkit for LLMs and VLMs, built around the TurboMind inference engine with an optional PyTorch backend. It quantizes KV cache to int8/int4 and weights to int4 (AWQ) — the combination that lets large models serve on modest GPUs.
It ships as a complete serving stack: an OpenAI-compatible API server, a CLI chat client, and an inference benchmark suite. Its mixed-precision acceleration work with TurboMind is published (EuroSys 2027) with an arXiv paper.
Hardware support spans NVIDIA GPUs from V100 upward (TurboMind MXFP4) and Huawei Ascend NPUs, with a supported-models matrix covering Qwen, InternVL and the wider open-model lineup.