• NVIDIA TensorRT LLM enables developers to build high-performance inference engines for large language models (LLMs), but deploying a new architecture traditionally requires significant manual effort. • To address this challenge, today we are announcing the availability of AutoDeploy as a beta feature in TensorRT LLM. • AutoDeploy compiles off-the-shelf PyTorch models into inference-optimized graphs. • This avoids the need to bake inference-specific optimizations directly into model code, reducing LLM deployment time. • AutoDeploy enables the shift from manually reimplementing and optimizing each model toward a compiler-driven workflow that separates model authoring from inference optimization. • This post introduces AutoDeploy architecture and capabilities and shows how it enabled support for recent NVIDIA Nemotron models at launch.
Article Summaries:
- NVIDIA has released AutoDeploy, a beta feature in TensorRT LLM that automates the conversion of off‑the‑shelf PyTorch models into inference‑optimized TensorRT graphs. AutoDeploy extracts a computation graph from a Hugging Face model and applies transformations such as sharding, quantization, KV‑cache insertion, attention fusion, and CUDA‑Graph optimizations, eliminating the need for manual reimplementation of inference logic. The tool supports over 100 text‑to‑text LLMs and early support for vision‑language and state‑space models, including NVIDIA’s Nemotron 3 Nano and the Llama family. AutoDeploy enables rapid deployment at launch with a clear path for incremental performance improvements.
Sources: