Model Serving

High-throughput, memory-efficient LLM inference and serving engine with PagedAttention and continuous batching — built for production GPU workloads.

Official Python client for the Ollama local LLM runtime — pull, run, and chat with open-weight models on your machine.

Open-source Python framework for packaging, serving, and deploying ML/LLM models as production APIs — model-agnostic with built-in adaptive batching.

BentoML-built CLI and Python framework for self-hosting open-weight LLMs as OpenAI-compatible API endpoints with one command.