Running LLMs is currently fraught with friction: dependency hell and the ungoverned management of massive weight files (GGUF, safetensors).
In this talk, we introduce Docker Model Runner (DMR), an open initiative to bring the same standard of reproducibility to AI models that containers brought to code. We will explore how DMR streamlines the "pull, push, run, serve" lifecycle by treating AI models as first-class OCI (Open Container Initiative) artifacts, decoupling the model weights from the inference engine.
We will dive deep into the architecture of the Docker Model Runner, covering:
Models as Artifacts: How we package and distribute GGUF and safetensors formats using OCI-compliant registries, eliminating the need for arbitrary file downloads.
A Unified Interface: How DMR abstracts the complexity of underlying inference backends. We will discuss the integration of llama.cpp for broad hardware compatibility (CPU/GPU) and vLLM for high-performance production serving.
The Model Driver Pack: How the runner handles hardware acceleration automatically, managing the interface between the container runtime and host resources (NVIDIA CUDA, Vulkan, etc.).
Developer Experience: A look at the docker model CLI plugin and the local REST API that allows developers to swap models without changing their client code.
Join us to see how we are building a standardized, open ecosystem where docker model run ai/gemma3 is all you need to start building AI applications.
Key Takeaways for the Review Committee (Why this fits FOSDEM): Open Standards: The talk focuses on OCI compliance and open weights (GGUF/safetensors).
Open Source Integration: It highlights the usage and orchestration of popular open-source projects (llama.cpp and vLLM).
Technical Depth: It addresses infrastructure challenges (GPU passthrough, artifact management) rather than just high-level AI concepts.