PolyAI has launched Dialog-RSN-1, a dialog mannequin that perceives the caller’s audio straight as a substitute of studying a transcript. It fuses turn-taking, speech recognition, operate calling and response era into one audio-native mannequin, and is already dealing with reside manufacturing calls.
Key Takeaways
- Dialog-RSN-1 is audio-aware on the enter facet solely; TTS stays separate, so the output voice stays controllable.
- It runs as a request-based LLM probed on demand, not an always-on stream that pins a GPU.
- Flip-taking is the mannequin’s first output token:
EMPTY,ONGOINGorCOMPLETE. - PolyAI experiences sub-300ms responses, +11% relative containment at a restaurant group, and −37% latency at an insurer.
- English solely at launch, delivered by means of PolyAI’s platform fairly than open weights or a public API.
Is it deployable, and by whom?
Sure, however solely by means of PolyAI: no open weights, no public API but. Current clients can allow it in the present day; new clients can request early entry.
- Firm stage: giant, high-call-volume enterprises. PolyAI experiences 100+ enterprise clients and a couple of,000+ reside deployments at its $86M Sequence D in December 2025. Self-serve builders and SMBs are usually not the goal.
- Industries: eating places, insurance coverage, monetary providers, healthcare, resorts, retail, telecom, journey and utilities.
- Purposes: reserving and reservations, billing and funds, authentication, name routing, order administration and troubleshooting.
The structure
Two architectures dominate, and every concedes one thing. A cascaded stack sends solely the ASR’s greatest guess to the LLM, so tone, hesitation and recognition uncertainty are gone earlier than the LLM sees something. Tuning means hand-adjusting end-pointing parameters and ASR biasing that not often generalize throughout use instances. Speech-to-speech fashions resembling GPT Realtime and Gemini Dwell hold the audio however bake the voice into the mannequin, limiting pronunciation management, and always-on full-duplex variants pin a GPU for your entire name.
Dialog-RSN-1 is audio-aware on enter solely: one mannequin causes over uncooked audio and fingers era to a separate, promptable TTS system. It’s probed on demand fairly than streamed: a high-recall VAD plus just a few timers resolve when to run it, and the primary token of the reply settles whether or not the agent ought to converse. Low cost acoustic cues solely select when to ask; the mannequin, with full context, makes the precise turn-taking name.
The way it was constructed
PolyAI post-trained open-weight multimodal fashions with supervised and reinforcement finetuning on in-house information. The pipeline is broadly base-model agnostic; PolyAI evaluated Gemma, GPT-OSS, Qwen and Mistral. Concentrating on sub-300ms on A100 GPUs places candidates within the 8B dense to 30B sparse vary. Latency work contains prefilling the eye cache whereas the consumer speaks, an append-only immediate template to reduce cache invalidation, routing every caller to the identical GPU, a finetuned speculative drafter with imply acceptance of three.9 tokens, and auto-reasoning discovered throughout RFT. Transcription runs final, after the response or software name, in parallel with speech era.
Outcomes
PolyAI evaluated on Dialog-Eval, an inner benchmark it plans to open-source. Every instance is a name truncated at one resolution level, scoring a single atomic subsequent step fairly than a full rollout. PolyAI experiences Dialog-RSN-1 because the highest-scoring real-time succesful mannequin, places the cascaded Audio-score ceiling close to 77, and notes GPT Realtime 2.1 scoring on par with cascades on audio-aware examples. On transcription, gpt-4o-transcribe’s WER improved from 7.8% to six.9% as soon as given the identical context, with Dialog-RSN-1 decrease nonetheless.
For this launch PolyAI targeted on English; Raven 3.5 stays its advice for non-English and wealthy internet chat. A technical report and a Dialog-Eval paper are deliberate.
Try the Technical particulars right here. Additionally, be happy to comply with us on Twitter and don’t neglect to affix our 150k+ML SubReddit and Subscribe to our Publication. Wait! are you on telegram? now you’ll be able to be a part of us on telegram as nicely.
Have to companion with us for selling your GitHub Repo OR Hugging Face Web page OR Product Launch OR Webinar and many others.? Join with us
