lmdeploy
InternLM/lmdeploy · Python · Apache-2.0
Turbocharge LLM/VLM serving with 4-bit weights and a fast gRPC API.
この解説はまだ日本語がありません。英語の原文を表示しています。
LMDeploy combines a TurboMind inference backend with weight quantization, continuous batching and an OpenAI-compatible server, tuned for throughput on a single consumer GPU.
スコア
導入の要点
pip install lmdeploy 深読み
LMDeploy is InternLM's deployment toolkit for LLMs and VLMs, built around the TurboMind inference engine with an optional PyTorch backend. It quantizes KV cache to int8/int4 and weights to int4 (AWQ) — the combination that lets large models serve on modest GPUs.
It ships as a complete serving stack: an OpenAI-compatible API server, a CLI chat client, and an inference benchmark suite. Its mixed-precision acceleration work with TurboMind is published (EuroSys 2027) with an arXiv paper.
Hardware support spans NVIDIA GPUs from V100 upward (TurboMind MXFP4) and Huawei Ascend NPUs, with a supported-models matrix covering Qwen, InternVL and the wider open-model lineup.