NVIDIA Audex Unified Audio-Text LLM Voice Agent Workflow

NVIDIA Audex (Nemotron-Labs-Audex-30B-A3B, released July 7, 2026) is a unified audio-text large language model that both understands and generates audio and speech through a single model. Unlike cascaded voice systems that chain separate ASR, LLM, and TTS models, Audex processes audio inputs directly into the text embedding space and treats audio outputs as text tokens. It is a 30B total parameter Mixture-of-Experts model with 3B activated parameters per token, built on the Nemotron-Cascade-2-30B-A3B text backbone. This backbone is a hybrid Mamba-Transformer with 52 layers, 128 routable experts, and 6 activated experts, optimized through multi-stage supervised fine-tuning plus text-only Cascade RL to avoid the multimodal text regression that plagues other audio-language models. A smaller Audex-2B model is also available. Audex is released under a non-commercial license. It is among the strongest open models for audio-text tasks and the only one that generates general audio beyond speech. It leads Qwen3.5-35B-A3B on several reasoning and alignment tasks.

BUSINESS PROBLEM

Voice AI systems today require minimum three separate models: Whisper or similar for ASR, GPT/Claude for reasoning, and ElevenLabs or similar for TTS. Each stage adds latency (typically 800ms-2s total), compounds errors (ASR errors propagate through LLM reasoning to TTS output), and requires separate infrastructure management. According to NVIDIA's Audex technical release (July 2026), cascaded systems lose an estimated 15-25% of semantic content between the ASR and LLM stages for accented speech, technical terminology, and code-switching. Each model also has independent hosting costs: ASR at $0.006/min, LLM at $0.01-0.03 per query, TTS at $0.015/min. A contact center processing 100,000 call minutes per month pays $1,000-3,000+ for cascaded voice AI infrastructure. Audex eliminates the cascade entirely with one unified model that keeps audio in its native representation throughout the reasoning process. It preserves the text intelligence of its Nemotron backbone while adding native audio understanding and generation, making it ideal for developers who need voice capability without sacrificing text reasoning quality.

WHO BENEFITS

Voice AI researcher building speech-to-speech applications who is frustrated by the compounding errors in cascaded ASR-LLM-TTS architectures and wants a single model that handles audio end-to-end. Embedded systems engineer deploying voice AI on resource-constrained hardware who cannot afford separate ASR, LLM, and TTS model footprints. Accessibility software developer building screen reader or voice control applications for users with accented speech who needs a model that does not degrade on non-standard pronunciation.

HOW IT WORKS

Step 1 - Model Download. Download Audex-30B-A3B or Audex-2B from Hugging Face under the non-commercial license. Step 2 - Backend Setup. Deploy with Megatron-LM for training or vLLM for optimized inference with BF16 precision. Step 3 - Audio Input. Audio inputs enter directly into the text embedding space via the audio encoder, bypassing separate ASR. Step 4 - Unified Reasoning. The MoE model reasons across audio and text with 3B activated parameters per token through the hybrid Mamba-Transformer architecture. Step 5 - Audio Output. The model generates audio output tokens natively, treated as regular tokens in the output sequence. Step 6 - Tool Integration. Function calls and MCP tools are represented as text tokens within the unified sequence, enabling agentic voice workflows. Step 7 - Multi-Turn Conversation. Maintain audio context across conversation turns with the 30B model's capacity for long-form interaction. Step 8 - Deployment. Serve with vLLM for production inference with configurable latency/throughput trade-offs.

TOOL INTEGRATION

NVIDIA Audex 30B-A3B (Nemotron-Labs, July 2026, non-commercial) - Unified audio-text LLM. Audex-2B (Nemotron-Labs) - Smaller model for resource-constrained deployment. Nemotron-Cascade-2-30B-A3B - Text backbone with hybrid Mamba-Transformer architecture. Megatron-LM (NVIDIA) - Training and fine-tuning framework. vLLM - Optimized inference serving with BF16 support. Hugging Face - Model weights and quickstart implementations. Audio encoder - Direct audio-to-embedding mapping. Text-to-audio decoder - Native audio output generation. Cascade RL - Text-only reinforcement learning to preserve text intelligence.

ROI METRICS

Cascaded infrastructure eliminated: single model replaces ASR + LLM + TTS pipeline. Semantic accuracy improvement: estimated 15-25% better content preservation for accented speech vs cascaded systems. Text intelligence preserved: matches Nemotron-Cascade-2 text backbone with marginal regression (per NVIDIA's benchmarks). Leads Qwen3.5-35B-A3B on reasoning and alignment tasks. Audex-2B enables deployment on consumer-grade hardware for lighter workloads. Non-commercial license allows research and evaluation. Single model serving reduces infrastructure complexity and hosting costs vs triple-model cascade. Multi-stage SFT plus Cascade RL prevents the multimodal text regression common in other audio-language models.

CAVEATS

HIGH - Non-commercial license restricts production and commercial use; evaluate licensing terms before building production systems. MEDIUM - 30B-A3B requires significant GPU resources for inference; 8GB+ VRAM minimum, 24GB+ recommended for production throughput. MODERATE - Early release (July 2026); community tooling, documentation, and best practices are still developing. MEDIUM - Audio generation quality for general audio (not speech) is a new capability; practical quality benchmarks for music and sound effects are not yet established across the community.