Release Cadence and Community Contributions
- Release Schedule: TensorRT Edge-LLM aims for a monthly release cadence to provide consistent updates and new features.
- Contributions: We welcome community contributions. PRs will be reviewed and integrated into upcoming releases.
Expanding Model Support
To support the needs of Physical AI, the list of models is continuously expanding to address limitations in operator support, multimodal processing, and runtime compatibility. As model providers release more advanced architectures, support for legacy models may be deprecated to prioritize performance and modern standards.
Models and Features in the Roadmap:
Nemotron v2 (0.6.0)Qwen3(+VL)-30B-A3B (low-precision MoE) (0.6.0 - 0.8.0)Qwen3-Omni/TTS/ASR (0.9.1 full support)Qwen3.5/3.6 (Dense 0.7.0, MoE 0.8.0)Alpamayo 1 (0.7.1)- Gemma 3n/
Gemma4 (0.9.1)
- Cosmos 3
- Native Video Processing
Inference System and Performance Optimizations
TensorRT Edge-LLM focuses on continuous code refactoring and optimization to ensure models run efficiently on target devices. Development goals include minimizing latency and memory footprint while maintaining a modular C++ runtime.
Optimization Roadmap:
EAGLE3 Improvements: Implementing Draft and Base embedding sharing. (0.5.0)KV Cache Compression: FP8 KV Cache integration. (0.5.0)Architectural Refinement: Unifying Vanilla and EAGLE Runtimes. (0.7.0)Kernel Exploration: Exploring Python DSL-based kernels (0.6.0), and performance improvements of the kernels- Agentic Workflow Integration: Exploring agentic workflow support
Platform Support and Ease of Use
Efforts are focused on providing a robust developer workflow and broader hardware compatibility.
Platform and Usability Roadmap:
Environment Support: Compatibility for Jetpack 6.2 and Jetpack 6.2.1. (0.5.0)Workflow Optimization: Reducing GPU memory requirements for quantization and supporting direct integration with the HuggingFace Hub. (0.7.0 new loader, 0.8.0 check in)- OpenAI Compatible Server: Implemented High Level One-line Python API and OpenAI Compatible Server (0.7.0 Experimental)
- Deployment Tools: Recommended Dockerfiles for environment alignment and GPU hardware guidelines for model quantization and export.
Release Cadence and Community Contributions
Expanding Model Support
To support the needs of Physical AI, the list of models is continuously expanding to address limitations in operator support, multimodal processing, and runtime compatibility. As model providers release more advanced architectures, support for legacy models may be deprecated to prioritize performance and modern standards.
Models and Features in the Roadmap:
Nemotron v2(0.6.0)Qwen3(+VL)-30B-A3B (low-precision MoE)(0.6.0 - 0.8.0)Qwen3-Omni/TTS/ASR(0.9.1 full support)Qwen3.5/3.6(Dense 0.7.0, MoE 0.8.0)Alpamayo 1(0.7.1)Gemma4(0.9.1)Inference System and Performance Optimizations
TensorRT Edge-LLM focuses on continuous code refactoring and optimization to ensure models run efficiently on target devices. Development goals include minimizing latency and memory footprint while maintaining a modular C++ runtime.
Optimization Roadmap:
EAGLE3 Improvements: Implementing Draft and Base embedding sharing.(0.5.0)KV Cache Compression: FP8 KV Cache integration.(0.5.0)Architectural Refinement: Unifying Vanilla and EAGLE Runtimes.(0.7.0)Kernel Exploration: Exploring Python DSL-based kernels(0.6.0), and performance improvements of the kernelsPlatform Support and Ease of Use
Efforts are focused on providing a robust developer workflow and broader hardware compatibility.
Platform and Usability Roadmap:
Environment Support: Compatibility for Jetpack 6.2 and Jetpack 6.2.1.(0.5.0)Workflow Optimization: Reducing GPU memory requirements for quantization and supporting direct integration with the HuggingFace Hub.(0.7.0 new loader, 0.8.0 check in)