litellm
BerriAI/litellm · Python · Apache-2.0
One import to talk to OpenAI, Anthropic, Google, Mistral and 100+ more.
LiteLLM normalises every provider behind the OpenAI SDK, adds retries, fallbacks, load balancing, caching and a proxy server with spend tracking — so swapping models is a config change.
Score
Setup at a glance
pip install litellm Deep dive
LiteLLM puts 100+ LLMs behind one OpenAI-compatible interface: swap providers by changing a model string, and call chat, embeddings, images, audio and agent protocols from the same SDK instead of maintaining provider-specific client code.
As a production gateway it adds what teams actually need — virtual keys, per-team spend tracking, guardrails, load balancing and fallbacks — with published benchmarks showing 8ms P95 latency at 1k RPS.
Beyond chat it speaks the agent stack: MCP servers and A2A agents mount behind the same gateway, turning it into a single control plane for both model and tool access.