NVIDIA has launched Audex (Nemotron-Labs-Audex-30B-A3B), a unified audio-text giant language mannequin. It understands and generates each audio and speech. It additionally retains the textual content intelligence of its spine. The checkpoints, together with a smaller Audex-2B, are launched below a noncommercial license.
Most multimodal fashions pay a textual content tax. When labs add audio or imaginative and prescient output, textual content benchmarks usually drop. NVIDIA analysis workforce reviews this even for speech-only output fashions. Audex is designed to keep away from that regression.
TL;DR
- Audex is a single 30B-A3B MoE mannequin that handles audio out and in.
- Audio inputs enter the textual content embedding area; audio outputs are handled like textual content tokens.
- Textual content scores match the spine, with small beneficial properties and small losses per benchmark.
- Multi-stage SFT plus text-only Cascade RL avoids the same old multimodal textual content regression.
- It’s few of the open fashions that generate basic audio past speech.
What’s Audex?
Audex is a single Combination-of-Specialists (MoE) Transformer decoder. It has 30B whole parameters and 3B activated per token. The spine is Nemotron-Cascade-2-30B-A3B, a text-only MoE LLM. That spine is a hybrid Mamba-Transformer with 52 layers. It makes use of 128 routable consultants and 6 activated consultants.
The design is intentionally easy. Audio inputs are encoded and projected into the textual content embedding area. Textual content tokens and quantized audio tokens are then handled uniformly throughout era. There isn’t a thinker-talker cut up and no stacked cascade of fashions.
As a result of the design stays easy, Audex runs on commonplace LLM stacks. These embody Megatron-LM for coaching and vLLM for inference. It helps each an instruct mode and a considering mode. Context size reaches 1M tokens.
How the Unified Design Works
Three elements sit across the LLM spine:
- An audio encoder reads sound. Audex makes use of AF-Whisper from Audio Flamingo 3. It shares the Whisper Giant-v3 structure and handles 16kHz enter.
- Two-layer MLP adapters map audio options into the mannequin dimension.
- An prolonged vocabulary holds discrete audio output tokens. The unique 131,072 tokens develop to 205,312.
Audex makes use of two codecs for output. Speech makes use of X-Codec2 at 50 tokens per second. It applies single-layer finite scalar quantization (FSQ) with a 65,536 codebook.
Non-speech sound makes use of X-Codec at 200 tokens per second. It makes use of 4 flattened residual vector quantization (RVQ) layers. Advanced sound will get a bigger token funds than speech. The interactive demo under computes these token counts for any length.
” fashion=”width:100%;border:0;show:block;top:600px;overflow:hidden” scrolling=”no” loading=”lazy” title=”Audex interactive demo”>
Coaching
Audex wants no audio pretraining. It begins from the text-only SFT checkpoint. Coaching then provides capabilities stage by stage.
The multi-stage SFT curriculum runs so as: textual content SFT, audio warmup, audio era, then audio understanding. Throughout audio warmup, textual content token embeddings keep frozen. Unfreezing them degraded textual content high quality in ablations.
NVIDIA analysis workforce additionally examined a single-stage recipe that mixes all information without delay. That recipe broke long-context retrieval on NIAH. Multi-stage coaching averted this, so it turned the default.
After SFT, the analysis workforce applies text-only Cascade RL and multi-domain on-policy distillation (MOPD). Audio duties present marginal or no regression after this text-only RL. Textual content scores enhance on the similar time.
The information combine is giant. It combines 157.4B audio tokens and 320.5B textual content tokens. Duties span ASR, AST, TTS, text-to-audio, and audio understanding.
Benchmark and Efficiency
On textual content, Audex tracks its spine carefully. It scores 86.4 on MMLU-Redux in opposition to the spine’s 86.3. It even leads on IMO AnswerBench, 81.1 versus 79.3. Small drops seem on MMLU-Professional and GPQA-Diamond.
Audex additionally tops the text-only Qwen3.5-35B-A3B on a number of reasoning, alignment, and instruction-following benchmarks. The comparably sized Qwen3-Omni-30B-A3B-Considering exhibits giant reasoning drops versus its personal spine.
| Benchmark | Audex 30B-A3B | Qwen3.5-35B-A3B | Qwen3-Omni-30B-A3B-Considering |
|---|---|---|---|
| HMMT Feb25 | 92.2 | 89.0 | 60.4 |
| IMO AnswerBench | 81.1 | 74.8 | 59.9 |
| LiveCodeBench v6 | 85.3 | 74.6 | 59.2 |
| ArenaHard v2 | 81.6 | 65.4 | 55.1 |
| IFBench (immediate) | 77.8 | 70.2 | 52.4 |
On speech recognition, Audex leads these open fashions. It information 6.82 common phrase error fee on the OpenASR leaderboard. That beats Step-Audio-R1.1-33B and Qwen3-Omni-30B-A3B-Considering.
On audio understanding the image is combined. Audex leads open fashions on MMAU. It exhibits gaps on MMAR and MMSU versus the strongest audio LLMs. Audex additionally generates basic audio, which the opposite main open fashions can not.
| Audio benchmark | Audex 30B-A3B | Step-Audio-R1.1-33B | Qwen3-Omni-30B-A3B-Considering |
|---|---|---|---|
| MMAU | 75.6 | 73.6 | 75.4 |
| MMAR | 63.2 | 69.8 | 66.4 |
| MMSU | 63.4 | 74.1 | 70.2 |
| Audio Entailment | 95.0 | 61.6 | 61.6 |
| OpenASR (WER, decrease is healthier) | 6.82 | 7.91 | 8.00 |
| BigBenchAudio | 90.0 | 97.6 | not reported |
Audex leads on MMAU, Audio Entailment, and OpenASR phrase error fee. It trails these open baselines on MMAR, MMSU, and BigBenchAudio.
Use Instances with Examples
- Contemplate a multilingual name heart. Audex can transcribe a German name and translate it to English. Its speech translation output lists the supply language, transcript, then English translation.
- Contemplate accessibility tooling. A developer can add fixed-voice text-to-speech to a studying app. The Seed-TTS-Eval English phrase error fee is a low 1.70.
- Contemplate sound design or prototyping. A caption like “birds chirping in a forest” yields a 10-second clip. Basic audio era makes use of an enhancement VAE for 48kHz output.
- Contemplate a voice assistant. Speech-to-speech runs as a cascade, however one checkpoint serves each step. Audex scores 90.0 on BigBenchAudio.
Fast Begin Instance
Audex follows the ChatML template. The reference container is vLLM 0.20.0. Audio enter decoding wants the audio extras.
Audio understanding, ASR, and translation share one audio question-answering format. The
[
{
"id": "sample_0",
"sound": "/path/to/audio_0.wav",
"conversations": [
{"from": "human", "value": "nDescribe the audio in detail."},
{"from": "gpt", "value": "N/A"}
]
}
] The mannequin card ships a vLLM audio-QA script for this enter format.
# add audio codecs, then run audio QA offline
python3 -m pip set up "vllm"
python inference_scripts_vllm/audioqa_scripts/run_audioqa_vllm.py
--model-path "$(pwd)/checkpoint_folder_full"
--input-json ./inputs.json
--output-jsonl ./outcomes.jsonl
--tensor-parallel-size 8For audio understanding, the analysis workforce recommends top_p=0.9 and temperature=0.7. For recognition and translation, use grasping sampling. Technology duties want classifier-free steerage, proven within the demo’s recipe tab.
Strengths and Weaknesses
Strengths
- Marginal or no textual content regression versus its text-only spine.
- Single unified mannequin, appropriate with Megatron-LM and vLLM.
- Among the many strongest open fashions, solely Audex generates basic audio.
- Leads Qwen3.5-35B-A3B on a number of reasoning and alignment duties.
Weaknesses
- The NVIDIA OneWay Noncommercial License limits business use.
- Audio understanding exhibits gaps on MMAR and MMSU versus prime audio LLMs.
- Speech-to-speech is cascaded, not native full-duplex.
- Reinforcement studying is text-only; audio-text RL is future work.
Try the Paper and Mannequin Weights. Additionally, be at liberty to observe us on Twitter and don’t overlook to hitch our 150k+ML SubReddit and Subscribe to our Publication. Wait! are you on telegram? now you possibly can be a part of us on telegram as properly.
Must associate with us for selling your GitHub Repo OR Hugging Face Web page OR Product Launch OR Webinar and so on.? Join with us
