Audio and omni models
Video has a soundtrack, and much of its meaning is in it: speech, music, crowd noise. This session shows how sound becomes a log-Mel spectrogram and then audio tokens (Whisper-style encoders and neural codecs), how audio and video tokens share one clock in a single sequence (Qwen2.5-Omni's TMRoPE), and how an omni model reads text, images, audio and video and answers in text and speech. It ends with video tasks that need both senses: active speaker detection, audio events and commentary understanding.
What you will learn
- How a waveform becomes a Whisper-style log-Mel spectrogram: 16 kHz, 25 ms Hann window, 10 ms hop, 80 or 128 mel bins, log and normalization
- Whisper's encoder and decoder format (conv stem with stride 2, 1,500 frames per 30 s, task tokens, 20 ms timestamps) and how LLM audio encoders pool to 25 tokens per second
- Continuous audio tokens for understanding versus discrete codec tokens for speech generation; residual vector quantization and the bitrate formula (EnCodec, Mimi)
- How Qwen2.5-Omni aligns audio and video in time: TMRoPE with one time id per 40 ms, and time interleaving in 2-second chunks
- The Thinker–Talker design: why the speech decoder reads hidden states and text, and how a block-windowed DiT and a vocoder stream the waveform
- Active speaker detection with audio–visual synchrony and cross-attention (TalkNet), and how audio events and commentary help key-moment detection
- How to count the token cost of a video with sound and choose between a cascade and an end-to-end omni model
Why this matters
In session 183 we counted what images and video cost a language model. Video has a second stream that we have ignored so far: sound. A cooking video says the recipe out loud. In a football match, the crowd roars a second before the replay starts. In a meeting recording, the question “who said that?” can only be answered by matching a voice to a face. A vision-language model that is deaf misses all of this.
This session adds audio to the picture in four steps. First, how sound becomes a log-Mel spectrogram, a small image of frequency over time. Second, how a speech encoder in the style of Whisper turns that image into vectors, and how many audio tokens one second costs (continuous vectors, or discrete codes from a neural codec). Third, how audio and video are aligned in time inside one sequence, using Qwen2.5-Omni’s TMRoPE as the worked example. Fourth, omni models: one model that reads text, images, audio and video, and answers in text and in speech. We end with video applications that need both senses: active speaker detection, audio events and commentary understanding.
Audio is also a cost question. With the Qwen2.5-Omni settings, one second of sound costs 25 tokens, while one second of 640×360 video at 2 frames per second costs 299 tokens. So in a typical video, sound is cheap context with a lot of information per token. That makes it one of the best things to add to a video system.
An omni model on one page
An omni model is a single model that accepts several input types (text, image, audio, video) and produces more than one output type (here text and speech). Figure 1 shows the open model we use as the main example, Qwen2.5-Omni-7B (2025). Its design is called Thinker–Talker [Certain]. The Thinker is a normal decoder LLM with a vision encoder and an audio encoder in front of it; it reads everything and writes text. The Talker is a second, smaller decoder that reads the Thinker’s hidden states and writes discrete speech tokens. A third part, token2wav, turns those tokens into a waveform. All numbers in the figure are from the paper and the released config.json [Certain].
Most of this page explains the arrows on the left (how sound becomes 25 tokens per second, and how those tokens are placed next to video tokens in time). The right side, the Talker and the speech decoder, comes in the section on omni models.
Sound as a picture: the log-Mel spectrogram
A microphone records air pressure many times per second. Speech models resample everything to 16,000 samples per second (16 kHz). This is enough for speech: by the sampling theorem, 16 kHz keeps frequencies up to 8 kHz, and most of what makes speech understandable is below that [Certain]. But a raw waveform is a bad input for a Transformer: 30 seconds are 480,000 numbers, and the same word looks very different as a waveform when the pitch or the microphone changes.
So we first describe which frequencies are present at each moment. Cut the signal into short, overlapping pieces (frames), multiply each by a smooth window, and take a Fourier transform. This is the short-time Fourier transform (STFT). Its squared magnitude is the power spectrogram:
- Read the symbols
- \(x[n]\)
- the waveform, one number per sample (16,000 per second)
- \(N\)
- window length in samples: 400 samples = 25 ms in Whisper
- \(H\)
- hop (stride) between frames: 160 samples = 10 ms, so 100 frames per second
- \(w[n]\)
- the Hann window, \(w[n] = \tfrac12 - \tfrac12\cos(2\pi n/N)\); it fades each frame in and out so the edges do not create false frequencies
- \(k\)
- frequency bin; bin \(k\) is at \(k \cdot 16{,}000/400 = 40k\) Hz, so there are 201 bins from 0 to 8,000 Hz
- \(m\)
- frame index (time); frame \(m\) starts at \(mH\)
Human hearing does not treat all frequencies equally. We hear the step from 100 Hz to 200 Hz as large, and the step from 7,000 Hz to 7,100 Hz as almost nothing. The mel scale models this: it is close to linear below 1 kHz and logarithmic above. Whisper (through librosa) uses the Slaney version [Certain]:
We place \(B\) triangular filters at equal distances on the mel scale between 0 and 8 kHz (so they are narrow at low frequencies and wide at high ones), sum the power under each triangle, take the logarithm, and normalize. In Whisper’s code this is:
- Read the symbols
- \(F[b, k]\)
- weight of frequency bin \(k\) in mel filter \(b\) (a triangle, scaled so every filter has the same area)
- \(B\)
- number of mel filters: 80 in the original Whisper, 128 in Whisper large-v3 and in Qwen2-Audio and Qwen2.5-Omni [Certain]
- \(\max(M) - 8\)
- a floor 8 orders of magnitude (80 dB) below the loudest value, so silence does not become \(-10\)
- \(L\)
- the final input, roughly in \([-1, 1]\); the paper says the input is scaled to “between -1 and 1 with approximately zero mean” [Certain]
The result is a \(B \times T\) image: \(B\) mel bins tall and 100 columns per second wide. Whisper always pads or cuts audio to 30 seconds, so its input is \(128 \times 3{,}000\) (large-v3) [Certain]. We checked our filter bank against the one shipped with Whisper: the largest difference is about \(10^{-7}\).
How to read Figure 2
Start with the rising whistle: its frequency goes from 200 Hz to 4,000 Hz in one second, so at frame 50 (0.5 s) it is at 2,100 Hz. The readout shows the brightest mel bin and its center frequency (about 2,124 Hz with 128 mels, 2,042 Hz with 40 mels: fewer, wider filters mean a coarser answer). The line is straight at the bottom and bends at the top because the mel axis compresses high frequencies. Now pick the vowel: it is a stack of harmonics of 140 Hz, and the brightest band near 700 Hz is the first formant (a resonance of the mouth that tells “a” from “i”). The claps are short, wide-band bursts: vertical stripes. Speech is a mix of all three patterns, which is why one 2D image works so well.
Speech encoders: Whisper and its descendants
Whisper
Whisper (OpenAI, 2022) is an encoder-decoder Transformer trained on 680,000 hours of audio paired with transcripts found on the internet [Certain]. The labels are “weak” (not checked by hand), but there are so many of them that the model became robust to accents, noise and microphones. Five sizes were released, from Tiny (4 layers, width 384, 39M parameters) to Large (32 layers, width 1,280, 1,550M parameters) [Certain].
The encoder is short to describe [Certain]: a stem of two 1D convolutions over time (filter width 3, GELU), where the second has stride 2; then sinusoidal position embeddings; then pre-norm Transformer blocks; then a final LayerNorm. The mel bins are the input channels of the first convolution, so the convolution looks at 3 neighboring frames at a time across all frequencies. The stride halves the frame rate from 100 to 50 per second, so 30 s become 1,500 encoder frames of 20 ms each.
The decoder writes text with special tokens that set the task [Certain]: <|startoftranscript|>, then a language token (99 languages), then <|transcribe|> or <|translate|>, then either <|notimestamps|> or timestamp tokens. Timestamps are quantized to 20 ms, “the native time resolution of Whisper models”, which is exactly one encoder frame [Certain]. A <|nospeech|> token marks segments without speech. This is the same “special tokens as a program” idea as chat templates in session 9.
From Whisper to an LLM’s audio encoder
For an audio-language model, we keep only Whisper’s encoder and connect it to an LLM, like a vision encoder in session 177. Qwen2-Audio does this with Whisper large-v3 and adds an average pool with stride 2 after the Transformer, so each output vector covers about 40 ms [Certain]. Qwen2.5-Omni reuses that encoder (32 layers, width 1,280, 20 heads, 128 mels) and a linear projection to the LLM width of 3,584 [Certain]. In the released code, the number of tokens for \(T\) mel frames is:
- Read the symbols
- \(T\)
- number of mel frames (100 per second)
- \(T_1\)
- frames after the stride-2 convolution (50 per second, as in Whisper)
- \(T_{\text{audio}}\)
- tokens after the stride-2 average pool: 25 per second, one per 40 ms
\(T = 3{,}000\) mel frames. After the convolution: \(\lfloor 2{,}999/2 \rfloor + 1 = 1{,}500\) (Whisper’s encoder length). After pooling: \(\lfloor 1{,}498/2 \rfloor + 1 = 750\) tokens. A 45 s clip gives \(4{,}500 \to 2{,}250 \to 1{,}125\) tokens. One minute gives 1,500 tokens. People speak roughly 150 words per minute [Likely], so that is about 10 audio tokens per spoken word, several times more than the same words cost as text tokens.
Two more changes make the encoder streamable in Qwen2.5-Omni [Certain]. The paper changes the encoder “from full attention over the entire audio to performing attention in blocks of 2 seconds each”; in the code this is n_window = 100, so each block is 200 mel frames. And the encoder is initialized from Whisper large-v3 and then trained with the LLM. From the config, the encoder has about 635M parameters (our count). With 2-second blocks, the encoder can process audio as it arrives, block by block, which is needed for chunked prefill in a live conversation.
- input: waveform at any rate; LLM width \(d_{\text{LLM}}\)
- resample to 16 kHz; for Whisper itself, pad or cut to 30 s
- STFT with a 400-sample Hann window and a 160-sample hop; keep \(|\cdot|^2\) // (184.1), 100 frames/s
- apply 128 mel filters, \(\log_{10}\), clamp to \(\max(M) - 8\), then \((x + 4)/4\) // (184.2), (184.3)
- split the frames into blocks of 200 (2 s) // streaming variant only
- conv1d (3, stride 1) \(\rightarrow\) GELU \(\rightarrow\) conv1d (3, stride 2) \(\rightarrow\) GELU; add sinusoidal positions // 50 frames/s
- 32 pre-norm Transformer layers, attention only inside each block
- average pool, size 2, stride 2; LayerNorm; linear to \(d_{\text{LLM}}\) // (184.4), 25 tokens/s
- wrap with audio-start and audio-end tokens; return the audio tokens
Audio tokens: continuous vectors or discrete codes
“Audio token” means two different things in practice, and interviews often test whether you know the difference.
- Continuous audio tokens are the encoder output vectors above. They enter the LLM exactly like image tokens: through a projection, in place of placeholder tokens. They are good for understanding (speech recognition, audio questions), because the encoder keeps semantic and acoustic detail. But an LLM cannot generate them with a softmax, because they are not from a finite vocabulary.
- Discrete audio tokens are integers from a codebook. There are two families. Semantic units come from clustering self-supervised speech features (for example k-means on HuBERT features [Certain]); they keep what was said but lose much of the voice. Acoustic codec tokens come from a neural audio codec such as SoundStream, EnCodec or Mimi: an encoder-quantizer-decoder trained to compress and rebuild the waveform. Codec tokens are what a model generates when it speaks.
Neural codecs quantize with residual vector quantization (RVQ). The first codebook approximates the vector; the second codebook approximates what is left (the residual); and so on:
- Read the symbols
- \(\mathbf{z}\)
- one latent vector from the codec encoder (one time step)
- \(\mathbf{c}^{(m)}_j\)
- entry \(j\) of codebook \(m\); each codebook has \(K\) entries
- \(q_m\)
- the integer token chosen in codebook \(m\)
- \(\mathbf{r}_m\)
- the residual left after \(m\) codebooks; it shrinks as \(m\) grows
- \(N_q\)
- number of codebooks used; fewer codebooks = lower bitrate and lower quality
Each time step produces \(N_q\) integers of \(\log_2 K\) bits, so the bitrate and the number of tokens per second are:
EnCodec at 24 kHz has 75 frames per second (encoder strides 2, 4, 5, 8, so \(24{,}000/320 = 75\)) and up to 32 codebooks of 1,024 entries (10 bits) [Certain]. With 8 codebooks: \(75 \times 8 \times 10 = 6{,}000\) bits/s = 6 kbps and 600 tokens per second; with 2 codebooks, 1.5 kbps [Certain]. Kyutai’s Mimi codec (used by the Moshi dialogue model) runs at only 12.5 frames per second with 8 RVQ levels [Certain], which is 100 tokens per second and, with 2,048-entry codebooks, about 1.1 kbps [Likely]. Mimi also distills semantic information from a self-supervised speech model into its first level [Certain], so one codec mixes both token families.
Our code below runs RVQ with k-means codebooks of 64 entries on random 8-dimensional vectors. The relative error after 1, 2, 3, 4 codebooks is 0.645, 0.417, 0.272, 0.179: each extra codebook removes about a third of what is left.
Figure 3 compares the token cost of one clip in each representation, next to the video tokens of the same clip. For Qwen2.5-Omni the total for a video with sound is:
- Read the symbols
- \(S\)
- clip length in seconds
- \(f\)
- video sampling rate in frames per second; two frames form one temporal patch, as in Qwen2.5-VL (session 182)
- \(\hat H, \hat W\)
- frame size after rounding to multiples of 28 (patch 14 × merge 2); a 640×360 frame becomes 644×364, so \(13 \times 23 = 299\) tokens per frame pair
How to read Figure 3
At the default (60 s, 2 fps, 640×360), video costs 17,940 tokens and Omni audio 1,500: sound is only 7.7% of the 19,440 tokens. Switch to 1280×720 and the video jumps to 71,760 while the audio stays at 1,500. Now compare the audio bars: if an LLM had to read EnCodec codes it would need 36,000 tokens for one minute, 24 times the encoder vectors. That is why understanding uses continuous encoder tokens, and why speech generators use low-rate codecs (Mimi at 6,000 codes per minute) or predict several codebooks per step.
Aligning audio and video in time
When a video has sound, the model must know that the bang it hears at 4.0 s belongs to the frame at 4.0 s. There are two separate problems. Order: where do audio tokens go in the 1D sequence? Position: what position ids do they get, so that attention knows which audio and video tokens are close in time?
Positions: TMRoPE
Qwen2-VL introduced M-RoPE: each token has three position ids \((t, h, w)\) for time, row and column, and the rotary dimensions are split into three groups, one per id (session 14). In the 7B Thinker the 64 frequency pairs of each head are split as 16 for time, 24 for height and 24 for width [Certain]. Qwen2.5-Omni’s TMRoPE (time-aligned multimodal RoPE) fixes the meaning of the time id: one temporal id = 40 ms of real time, for audio and video alike [Certain] (position_id_per_seconds = 25 in the config). With a start id \(s_0\):
- Read the symbols
- \(s_0\)
- the first id of this video (one more than the largest id before it)
- \(j\) (audio)
- index of the audio token; token \(j\) covers \([0.04j,\, 0.04(j+1))\) seconds
- \(g\)
- index of the frame pair (temporal patch); it starts at \(g \cdot 2/f\) seconds
- \(f\)
- video frames per second; at \(f = 2\), consecutive pairs are 1 s apart, so their time ids differ by 25
- \(i, j\) (video)
- row and column of the merged 2×2 patch inside the frame
Audio gets the same id on all three axes, so for audio-to-audio attention TMRoPE is ordinary 1D RoPE. Video and audio that happen at the same moment get the same time id, so RoPE sees them as close, whatever their order in the sequence. Text after the video continues from the largest id + 1 [Certain].
Order: time interleaving in 2-second chunks
If we put all video tokens first and all audio last, the first audio token is thousands of positions away from the frame it belongs to, and a streaming model would have to wait for the whole video. Qwen2.5-Omni instead cuts the video-with-audio into chunks of 2 seconds of real time, and in each chunk places the video tokens first, then the audio tokens [Certain]. In the code a token belongs to chunk \(c\) when its time id satisfies
How to read Figure 4
The default (6 s, 2 fps, 2×2) gives 174 tokens in the order V8 A50 V8 A50 V8 A50: two frame pairs (8 video tokens) and 50 audio tokens per chunk. Click the first video token of the second row: its time id is 50 (2.0 s), and the audio token at 2.0 s, 8 positions later, has the same time id. Switch to 4 fps: four pairs per chunk (V16), time ids 0, 12, 25, 37, … (the floor in (184.8)). Switch to 0.5 fps with 8 s: some chunks have no frame at all, and the audio simply continues. In every case the text after the video starts at id 150 for 6 s, which is “largest id + 1”.
- input: video tokens grouped by frame pair \(g\) with merged grid \(h \times w\); audio tokens \(0 \dots A-1\); fps \(f\); start id \(s_0\)
- for each video token: \(t \leftarrow s_0 + \lfloor 25 g \cdot 2/f \rfloor\), \(h \leftarrow s_0 + i\), \(w \leftarrow s_0 + j\) // (184.8)
- for each audio token \(j\): \(t = h = w \leftarrow s_0 + j\)
- for chunk \(c = 0, 1, \dots\): append the video tokens with \(t - s_0 \in [50c, 50c + 50)\), then the audio tokens in the same range // (184.9)
- next id for text \(\leftarrow 1 + \max(\text{ids used})\)
- apply RoPE with the frequency pairs split (16, 24, 24) among \(t, h, w\)
- return the reordered tokens and their \((t, h, w)\) ids
Other systems solve the same problem differently. Many video-LLMs drop audio entirely, or run a separate speech recognizer and insert the transcript with timestamps as text (simple, but it loses tone, music and sound events). Some interleave per frame instead of per chunk. The trade-off is the same as in session 179: interleaving in one sequence gives full attention between modalities at the cost of a long sequence.
Omni models: text and speech out
The cascade and why it is not enough
The classic voice assistant is a cascade: a speech recognizer (ASR) writes the user’s words as text, an LLM answers in text, and a text-to-speech system (TTS) reads the answer aloud. It is modular and easy to debug. But it has two problems. It loses everything that is not words: tone, emotion, laughter, background sound. And it is slow: each stage waits for the previous one. Moshi’s authors describe the latency of such pipelines as “several seconds”, while Moshi itself reaches 160 ms in theory and 200 ms in practice [Certain].
Thinker and Talker
Qwen2.5-Omni keeps the strong text LLM and adds a mouth [Certain]. The Thinker (initialized from Qwen2.5-7B: 28 layers, width 3,584) generates text as usual. The Talker is a separate decoder (24 layers, width 896, 12 heads, from the config) with a vocabulary of 8,448 tokens that include the speech codes of a new codec called qwen-tts-tokenizer [Certain]. The Talker is “dual-track”, inspired by Mini-Omni: at each step it receives both the Thinker’s high-level hidden states and the embeddings of the text tokens the Thinker sampled, and shares the Thinker’s full context [Certain].
Why both? The paper gives two reasons [Certain]. Hidden states carry tone and attitude before the whole sentence is written, which a streaming voice needs. But hidden states encode meaning, not sound: two words with similar meaning but different pronunciation can have similar hidden states, so the sampled text tokens are needed to remove that ambiguity. The speech does not need word-level timing alignment with the text, which simplifies training data [Certain].
From codes to waveform, in a stream
The token2wav part has two models [Certain]: a DiT (diffusion Transformer, 22 layers, width 1,024) trained with flow matching turns codes into an 80-bin mel spectrogram, and a modified BigVGAN vocoder turns the mel spectrogram into a waveform (see session 229 and session 228). From the config, each code is repeated twice before the DiT, and BigVGAN upsamples each mel frame by \(5 \cdot 3 \cdot 2 \cdot 2 \cdot 2 \cdot 2 = 240\) samples, so one code is 480 samples; at a 24 kHz output rate that is 50 codes per second [Likely].
To stream, the DiT must not look at the whole future. Codes are grouped into blocks, and each block attends only to a window: 2 blocks back, itself, and 1 block ahead, a receptive field of 4 blocks [Certain]:
- Read the symbols
- \(u, v\)
- positions in the DiT input (query and key)
- \(b(\cdot)\)
- the block a position belongs to
- \(A_{uv}\)
- 1 if \(u\) may attend to \(v\)
So the waveform of block \(b\) can be produced as soon as block \(b+1\) of codes exists: the delay is one block, not the whole answer. Together with 2-second blocks in the audio encoder (chunked prefill) and a streaming Talker, this is how the first audio packet arrives quickly. The paper lists four latency sources: input processing, time from first text to first speech token, turning the first speech segment into audio, and the model’s own size and FLOPs [Certain].
Training and results
Pretraining has three stages [Certain]: (1) freeze the LLM and train the vision and audio encoders (adapters first); (2) unfreeze everything and add 800B tokens of image and video data, 300B of audio data and 100B of video-with-audio data, with sequences up to 8,192; (3) extend to 32k tokens with long audio and video. The Talker is trained in three more stages: in-context speech continuation, then DPO with rewards based on word error rate and pause errors to make speech generation stable, then speaker fine-tuning for specific voices [Certain]. On LibriSpeech, Qwen2.5-Omni-7B reports a word error rate of 1.8% (test-clean) and 3.4% (test-other), next to 1.8% and 3.6% for Whisper large-v3 in the same table [Certain]. So adding vision and speech output did not cost it speech recognition quality.
| Design | Example | Speech in | Speech out |
|---|---|---|---|
| Cascade | ASR → LLM → TTS | text transcript only | separate TTS after the text |
| Audio encoder + LLM | Qwen2-Audio | continuous encoder tokens | none (text only) |
| Thinker–Talker | Qwen2.5-Omni, Mini-Omni (text-guided speech) | continuous encoder tokens | codec tokens from a second decoder, streamed |
| Full duplex, parallel streams | Moshi (Mimi codec) | codec tokens of the user’s stream | codec tokens of its own stream, with “inner monologue” text first |
Applications: speakers, sound events and commentary
Active speaker detection
Active speaker detection (ASD) answers, for every face in every frame, “is this person speaking now?”. It is needed for meeting transcripts with names, for auto-framing in video calls, for dubbing, and for “who said X?” questions in Video-RAG. The naive rule “the lips move, so the person speaks” fails: people chew, smile and mouth along, and in dubbed films the lips do not match the voice. The reliable signal is synchrony: the mouth opens when the voice gets louder, at the same moment.
A simple synchrony score correlates the audio envelope \(a\) with the mouth-opening signal \(m_k\) of face \(k\) in a window around frame \(t\), after shifting the audio by a lag \(\delta\):
- Read the symbols
- \(a_t\)
- loudness of the audio at video frame \(t\) (25 frames per second, 40 ms each)
- \(m_{k,t}\)
- how open the mouth of face \(k\) is at frame \(t\)
- \(W\)
- window length in frames; longer windows give more evidence but blur speaker changes
- \(\delta\)
- audio-video offset in frames; a whole file is often a few frames out of sync
- \(\hat\delta\)
- the offset that best explains all faces, estimated once per clip
TalkNet (2021) learns this instead of hand-coding it [Certain]. A visual temporal encoder reads 112×112 face crops (receptive field 21 frames, 840 ms at 25 fps); an audio temporal encoder reads 13-dimensional MFCC features (receptive field 189 audio frames, 1,890 ms). Then cross-attention in both directions lets each stream query the other:
Here the queries come from one modality and the keys and values from the other (see session 20). Because attention can look a few frames to either side, it absorbs small offsets that a fixed frame-by-frame comparison cannot. A self-attention layer over the joined features then models the whole utterance, and a classifier labels each frame. TalkNet reached 92.3% mAP on the AVA-ActiveSpeaker validation set and 90.8% on its test set [Certain]. Its key ablation is about time: with a fixed input of 5 frames (0.2 s) it gets 75.2% mAP, with 25 frames (1 s) 87.9%, with 50 frames (2 s) 89.5% [Certain]. Removing the whole attention part drops it to 90.0% [Certain]. Figure 5 lets you feel both effects on a toy scene.
How to read Figure 5
Start with “mouth moves”: A and B are fine, but C is labeled as speaking whenever anyone talks, so C’s accuracy is 6.0% and the mean is 63.2%. Switch to sync: C drops out (95.6%) and the mean is 90.1%. Now move the window: 5 frames (0.2 s) is too little evidence for C (74.0%), 51 frames blurs the turn changes for A (80.8%); this is TalkNet’s long-context finding in miniature. Then set the audio delay to 3 frames (120 ms): sync breaks (mean 69.1%) because the mouth now leads the sound. Tick “estimate and correct the offset”: the lag search of (184.11) finds 3 and accuracy returns to 90.1%.
- input: video at 25 fps with audio; face detector and tracker
- detect and track faces; crop each face track to a fixed size (for example \(112 \times 112\), grayscale)
- compute audio features at a rate that is a multiple of the video rate (MFCC or log-Mel at 100 frames/s: \(100 / 25 = 4\) per video frame)
- estimate one global offset \(\hat\delta\) by maximizing audio–mouth agreement over the clip; shift the audio // (184.11)
- for each face track: encode the visual stream and the audio stream over a long window (1 to 2 s or the whole track)
- fuse with cross-attention in both directions, then self-attention over time // (184.12)
- classify each frame as speaking / not speaking; smooth the labels (for example a median filter over a few frames)
- return per-frame, per-face speaking scores; join with the ASR transcript by time to name the speaker of each sentence
Audio events and commentary understanding
Audio events are sounds that are not speech: a goal horn, glass breaking, a siren, applause, music. The standard dataset is Google’s AudioSet, with about 2 million 10-second YouTube clips labeled with 527 classes [Likely]. In video systems, audio events are cheap and strong triggers: a sudden crowd roar or a whistle often marks a key moment before anything clear happens on screen, and a gunshot or scream is a strong moderation signal. An omni model can do this zero-shot (“list the sounds and their times”), and with TMRoPE it can answer with times in seconds.
Commentary understanding combines three audio signals. The words of the commentator (“what a save!”) name the event; the prosody (pitch and loudness rising) tells how important it is; and the crowd noise confirms it. A cascade with ASR keeps only the words. A sports highlight system built on an omni model, or on an audio encoder plus a video encoder, can use all three. A practical recipe: compute a loudness and pitch track for the whole match (very cheap), run the omni model only on windows around the peaks, and ask it to describe the event and give its start time.
Code: the key mechanisms in one file
This script builds the Whisper-style log-Mel front end (184.1 to 184.3), a tiny audio encoder with the same shape rules as Qwen2.5-Omni’s (184.4), the TMRoPE ids with time interleaving (184.8, 184.9), and residual vector quantization (184.5). It runs on CPU in under a second. Our mel filter bank matches the one shipped with Whisper to about \(10^{-7}\).
import math, torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) SR, N_FFT, HOP = 16000, 400, 160 # 16 kHz, 25 ms window, 10 ms hop def mel(f): # Slaney mel scale (184.2), as in librosa / Whisper return 3 * f / 200 if f < 1000 else 15 + 27 * math.log(f / 1000) / math.log(6.4) def hz(m): return 200 * m / 3 if m < 15 else 1000 * math.exp((m - 15) * math.log(6.4) / 27) def mel_filters(n_mels, sr=SR, n_fft=N_FFT): # triangles, equally spaced in mel pts = torch.tensor([hz(m) for m in torch.linspace(0, mel(sr / 2), n_mels + 2).tolist()]) f = torch.linspace(0, sr / 2, n_fft // 2 + 1) lo, ce, hi = pts[:-2, None], pts[1:-1, None], pts[2:, None] tri = torch.clamp(torch.minimum((f - lo) / (ce - lo), (hi - f) / (hi - ce)), min=0) return tri * 2 / (hi - lo) # equal area per filter def log_mel(x, n_mels=128): # Whisper-style log-Mel (184.1)-(184.3) S = torch.stft(x, N_FFT, HOP, window=torch.hann_window(N_FFT), return_complex=True) S = S[..., :-1].abs() ** 2 # drop the last frame: 100 frames per second m = torch.clamp(mel_filters(n_mels) @ S, min=1e-10).log10() m = torch.maximum(m, m.max() - 8.0) # keep an 80 dB range return (m + 4.0) / 4.0 class AudioEncoder(nn.Module): # conv stem + transformer + pool + adapter def __init__(s, n_mels=128, d=64, d_llm=96, layers=2): super().__init__() s.conv1 = nn.Conv1d(n_mels, d, 3, padding=1) s.conv2 = nn.Conv1d(d, d, 3, stride=2, padding=1) # 100 -> 50 frames/s layer = nn.TransformerEncoderLayer(d, 4, 4 * d, batch_first=True, norm_first=True) s.blocks = nn.TransformerEncoder(layer, layers, enable_nested_tensor=False) s.pool = nn.AvgPool1d(2, stride=2) # 50 -> 25 tokens/s s.proj = nn.Linear(d, d_llm) # to the LLM width def forward(s, mel): # mel: (B, n_mels, T) h = F.gelu(s.conv2(F.gelu(s.conv1(mel)))).transpose(1, 2) T, d = h.shape[1], h.shape[2] pos = torch.arange(T)[:, None] / 10000 ** (torch.arange(0, d, 2) / d) h = h + torch.cat([pos.sin(), pos.cos()], dim=1) # sinusoidal positions h = s.pool(s.blocks(h).transpose(1, 2)).transpose(1, 2) return s.proj(h) t = torch.arange(30 * SR) / SR # 30 s: a 220 Hz tone plus a little noise x = 0.5 * torch.sin(2 * math.pi * 220 * t) + 0.01 * torch.randn(t.shape) M = log_mel(x) tok = AudioEncoder()(M[None]) print(tuple(M.shape), tuple(tok.shape), tok.shape[1] / 30) def tmrope_ids(seconds, fps=2, H=2, W=2, rate=25, chunk=2, tp=2, st=0): """(kind, t, h, w) for video-with-audio, time-interleaved in 2 s chunks (184.8), (184.9).""" n_grid = int(seconds * fps / tp) # one grid = tp frames vid = [("V", st + int(g * tp / fps * rate), st + i, st + j) for g in range(n_grid) for i in range(H) for j in range(W)] aud = [("A", st + k, st + k, st + k) for k in range(int(seconds * rate))] per, seq = chunk * rate, [] for c in range(math.ceil(seconds / chunk)): inside = lambda p: c * per <= p[1] - st < (c + 1) * per seq += [p for p in vid if inside(p)] + [p for p in aud if inside(p)] return seq seq = tmrope_ids(6) runs = [[seq[0][0], 0]] for p in seq: if p[0] == runs[-1][0]: runs[-1][1] += 1 else: runs.append([p[0], 1]) print(len(seq), " ".join(f"{k}{n}" for k, n in runs)) at4 = [i for i, p in enumerate(seq) if p[1] == 100] # temporal id 100 = 4.0 s print(at4, seq[at4[0]], seq[at4[-1]]) print("next text id:", max(max(p[1:]) for p in seq) + 1) def rvq(z, n_books=4, K=64, iters=10): # residual vector quantization (184.5) r, out = z.clone(), [] for m in range(n_books): C = r[torch.randperm(len(r))[:K]].clone() # k-means codebook on the residual for _ in range(iters): idx = torch.cdist(r, C).argmin(1) for k in range(K): if (idx == k).any(): C[k] = r[idx == k].mean(0) idx = torch.cdist(r, C).argmin(1) r = r - C[idx] out.append(round((r.norm() / z.norm()).item(), 3)) return out print(rvq(torch.randn(4096, 8))) # relative error after 1, 2, 3, 4 codebooks # Output: # (128, 3000) (1, 750, 96) 25.0 # 174 V8 A50 V8 A50 V8 A50 # [116, 117, 118, 119, 124] ('V', 100, 0, 0) ('A', 100, 100, 100) # next text id: 150 # [0.645, 0.417, 0.272, 0.179]
Three things to notice in the output. Thirty seconds become a \(128 \times 3{,}000\) spectrogram and then 750 tokens, 25 per second. In the interleaved sequence, the video tokens at 4.0 s (positions 116 to 119) and the audio token at 4.0 s (position 124) are 5 to 8 positions apart and share time id 100. And the text after a 6 s video starts at id 150. To run the real model, use Qwen2_5OmniForConditionalGeneration and Qwen2_5OmniProcessor in Hugging Face Transformers, with use_audio_in_video=True to interleave the soundtrack [Certain].
Where this is used
| Task | Audio part | Why audio helps |
|---|---|---|
| Subtitles, meeting notes, lecture search | Whisper-style ASR with 20 ms timestamps | The words are in the sound, not in the frames |
| “Who said what” in meetings, interviews, films | Active speaker detection (Figure 5) + ASR + face tracks | Joins voice to face by synchrony, not by lip motion alone |
| Key-moment detection, sports highlights | Loudness and crowd-noise peaks, commentator prosody, an omni model on the peak windows | Sound often marks the moment before the picture makes it clear |
| Video retrieval and Video-RAG | Transcript and audio-event tags with timestamps, indexed next to frame embeddings | Many queries are about what was said (“the part where she explains the API”) |
| Video classification and moderation | Audio-event classifier (gunshots, screams, music genre) | Cheap and robust to bad lighting or off-screen events |
| Video fingerprinting and near-duplicate detection | Spectrogram-based audio fingerprints | Re-uploads are often cropped or re-colored but keep the soundtrack |
| Voice assistants and live video chat | Omni model with streaming encoder and Talker | Low latency, and tone of voice is kept in both directions |
Common mistakes
- Wrong sample rate. Feeding 44.1 kHz or 48 kHz audio to a 16 kHz model without resampling shifts every frequency, and the model hears a different voice. Always resample to the processor’s
sampling_rate. - Mismatched mel settings. Whisper large-v3 and Omni use 128 mel bins; older Whisper models use 80. Features from the wrong extractor give garbage without any error message.
- Counting audio tokens with the wrong rate. Whisper’s encoder emits 50 frames per second; Qwen2-Audio and Qwen2.5-Omni emit 25 tokens per second after pooling; a codec at 75 Hz with 8 codebooks is 600 tokens per second if flattened. Use (184.4) and (184.6), not a guess.
- Dropping the soundtrack of a video. Many pipelines decode only the frames. For Qwen2.5-Omni you must pass
use_audio_in_video=True; otherwise the model answers from pixels alone. - Ignoring audio–video offset. Containers and re-encoding often shift audio by tens to hundreds of milliseconds. Sync-based methods break (Figure 5). Estimate the offset per clip, or use a model with enough temporal context to absorb it.
- Using lip motion as the speaking signal. Chewing, smiling and dubbed speech all move the lips. Use audio–visual synchrony and long windows.
- Treating continuous audio embeddings as generatable. An LLM can only sample from a finite vocabulary; to produce speech you need codec tokens and a decoder (or a separate TTS).
- Measuring only text quality of an omni model. Latency to the first audio packet, speech naturalness and voice stability are separate metrics; a model can be excellent in text and still sound robotic or slow.
Interview questions
How does Whisper turn a waveform into encoder input, and how many encoder frames is 30 seconds?basic
Why use a log-Mel spectrogram instead of the raw waveform?basic
How many LLM tokens does one minute of audio cost in Qwen2.5-Omni, and where does that number come from?core
Continuous audio embeddings vs discrete codec tokens: when do you use which?core
Explain residual vector quantization and compute EnCodec’s bitrate with 8 codebooks.core
What is TMRoPE, and why does Qwen2.5-Omni interleave audio and video in 2-second chunks?advanced
In the Thinker–Talker design, why does the Talker read the Thinker’s hidden states and not only the generated text?advanced
Where does latency come from in a spoken assistant, and how does an omni model reduce it?advanced
Design an active speaker detector for meeting videos. What features, what context, and what can go wrong?core
Cascade (ASR → LLM → TTS) or end-to-end omni model for a product? Give the trade-offs.core
You must find highlight moments in 90-minute football matches. How do you use audio, and what do you send to an omni model?advanced
Exercises
- How many Whisper encoder frames and how many Qwen2.5-Omni audio tokens does a 45-second clip produce? Use (184.4).
- A 10-second video at 1 fps with 640×360 frames and its soundtrack goes into Qwen2.5-Omni. How many video tokens, how many audio tokens, and what is the time-id step between consecutive frame pairs?
- EnCodec at 24 kHz with 4 codebooks: what is the bitrate, and how many codes does 10 s of audio produce?
- At 2 fps, in which 2-second chunk is the frame pair that starts at 3.0 s, and what is its time id (with \(s_0 = 0\))? Which audio token has the same time id?
- In Figure 5, the audio is 120 ms late. How many video frames is that at 25 fps, and why does a frame-by-frame correlation fail while TalkNet’s cross-attention can still work?
- Compare one minute of speech as Mimi codes and as EnCodec 6 kbps codes. How many tokens each, and what is the ratio?
Answerscheck
Key takeaways
- Audio enters models as a log-Mel spectrogram: 16 kHz, 25 ms windows, 10 ms hop, 80 or 128 mel bins, log and normalize (184.1 to 184.3). It is a \(B \times 100\)-per-second image.
- Whisper’s encoder (conv stem with stride 2, Transformer) gives 50 frames per second; LLM audio encoders (Qwen2-Audio, Qwen2.5-Omni) pool to 25 tokens per second (184.4). One minute = 1,500 tokens, far less than video.
- Continuous encoder tokens are for understanding; discrete codec tokens (RVQ, 184.5, 184.6) are for generating speech. Codec token rates vary a lot: 600/s for EnCodec 6 kbps, 100/s for Mimi.
- TMRoPE gives audio and video one shared clock (one id = 40 ms, 184.8) and interleaves them in 2-second chunks, video first (184.9).
- Thinker–Talker: a text LLM thinks, a second decoder speaks from its hidden states and text, and a block-windowed DiT (184.10) plus a vocoder stream the waveform.
- Audio–visual tasks need synchrony and long context: active speaker detection uses audio–mouth agreement (184.11) and cross-attention (184.12), not lip motion alone.
Key formulas
Next, in session 185 we put the pieces of this track together: a vision encoder, a connector and a small language model, trained end to end.
References for this session
- Radford et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision
- Xu et al. (2025). Qwen2.5-Omni Technical Report
- Tao et al. (2021). Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
- Radford et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision
- Xu et al. (2025). Qwen2.5-Omni Technical Report
- Tao et al. (2021). Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
- Chu et al. (2024). Qwen2-Audio Technical Report
- Défossez, Copet, Synnaeve and Adi (2022). High Fidelity Neural Audio Compression (EnCodec)
- Zeghidour et al. (2021). SoundStream: An End-to-End Neural Audio Codec
- Hsu et al. (2021). HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
- Défossez et al. (2024). Moshi: a speech-text foundation model for real-time dialogue
- Xie and Wu (2024). Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming
- Gemmeke et al. (2017). Audio Set: An ontology and human-labeled dataset for audio events
- Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
- Bai et al. (2025). Qwen2.5-VL Technical Report