LLM & VLM courseBlog
184Track 7: Vision and VLMs · session 22 of 38

Audio and omni models

Video has a soundtrack, and much of its meaning is in it: speech, music, crowd noise. This session shows how sound becomes a log-Mel spectrogram and then audio tokens (Whisper-style encoders and neural codecs), how audio and video tokens share one clock in a single sequence (Qwen2.5-Omni's TMRoPE), and how an omni model reads text, images, audio and video and answers in text and speech. It ends with video tasks that need both senses: active speaker detection, audio events and commentary understanding.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions on RoPE (14), cross-attention (20), anatomy of a VLM (177), case studies 2 (182) and visual token efficiency (183)

What you will learn

  • How a waveform becomes a Whisper-style log-Mel spectrogram: 16 kHz, 25 ms Hann window, 10 ms hop, 80 or 128 mel bins, log and normalization
  • Whisper's encoder and decoder format (conv stem with stride 2, 1,500 frames per 30 s, task tokens, 20 ms timestamps) and how LLM audio encoders pool to 25 tokens per second
  • Continuous audio tokens for understanding versus discrete codec tokens for speech generation; residual vector quantization and the bitrate formula (EnCodec, Mimi)
  • How Qwen2.5-Omni aligns audio and video in time: TMRoPE with one time id per 40 ms, and time interleaving in 2-second chunks
  • The Thinker–Talker design: why the speech decoder reads hidden states and text, and how a block-windowed DiT and a vocoder stream the waveform
  • Active speaker detection with audio–visual synchrony and cross-attention (TalkNet), and how audio events and commentary help key-moment detection
  • How to count the token cost of a video with sound and choose between a cascade and an end-to-end omni model

Why this matters

In session 183 we counted what images and video cost a language model. Video has a second stream that we have ignored so far: sound. A cooking video says the recipe out loud. In a football match, the crowd roars a second before the replay starts. In a meeting recording, the question “who said that?” can only be answered by matching a voice to a face. A vision-language model that is deaf misses all of this.

This session adds audio to the picture in four steps. First, how sound becomes a log-Mel spectrogram, a small image of frequency over time. Second, how a speech encoder in the style of Whisper turns that image into vectors, and how many audio tokens one second costs (continuous vectors, or discrete codes from a neural codec). Third, how audio and video are aligned in time inside one sequence, using Qwen2.5-Omni’s TMRoPE as the worked example. Fourth, omni models: one model that reads text, images, audio and video, and answers in text and in speech. We end with video applications that need both senses: active speaker detection, audio events and commentary understanding.

Audio is also a cost question. With the Qwen2.5-Omni settings, one second of sound costs 25 tokens, while one second of 640×360 video at 2 frames per second costs 299 tokens. So in a typical video, sound is cheap context with a lot of information per token. That makes it one of the best things to add to a video system.

An omni model on one page

An omni model is a single model that accepts several input types (text, image, audio, video) and produces more than one output type (here text and speech). Figure 1 shows the open model we use as the main example, Qwen2.5-Omni-7B (2025). Its design is called Thinker–Talker [Certain]. The Thinker is a normal decoder LLM with a vision encoder and an audio encoder in front of it; it reads everything and writes text. The Talker is a second, smaller decoder that reads the Thinker’s hidden states and writes discrete speech tokens. A third part, token2wav, turns those tokens into a waveform. All numbers in the figure are from the paper and the released config.json [Certain].

inputs and encodersThinker (reads, writes text)Talker and speech decoder
Figure 1. The Thinker–Talker design of Qwen2.5-Omni-7B. The labels under the encoders give the number of tokens each input becomes. The Talker receives the Thinker’s hidden states (width 3,584) and the text tokens it samples, and produces codec tokens; a DiT and a BigVGAN vocoder turn them into sound.

Most of this page explains the arrows on the left (how sound becomes 25 tokens per second, and how those tokens are placed next to video tokens in time). The right side, the Talker and the speech decoder, comes in the section on omni models.

Sound as a picture: the log-Mel spectrogram

A microphone records air pressure many times per second. Speech models resample everything to 16,000 samples per second (16 kHz). This is enough for speech: by the sampling theorem, 16 kHz keeps frequencies up to 8 kHz, and most of what makes speech understandable is below that [Certain]. But a raw waveform is a bad input for a Transformer: 30 seconds are 480,000 numbers, and the same word looks very different as a waveform when the pitch or the microphone changes.

So we first describe which frequencies are present at each moment. Cut the signal into short, overlapping pieces (frames), multiply each by a smooth window, and take a Fourier transform. This is the short-time Fourier transform (STFT). Its squared magnitude is the power spectrogram:

\[ P[k, m] = \left| \sum_{n=0}^{N-1} x[mH + n]\; w[n]\; e^{-2\pi i k n / N} \right|^2, \qquad k = 0, \dots, \tfrac{N}{2} \tag{184.1} \]
Read the symbols
\(x[n]\)
the waveform, one number per sample (16,000 per second)
\(N\)
window length in samples: 400 samples = 25 ms in Whisper
\(H\)
hop (stride) between frames: 160 samples = 10 ms, so 100 frames per second
\(w[n]\)
the Hann window, \(w[n] = \tfrac12 - \tfrac12\cos(2\pi n/N)\); it fades each frame in and out so the edges do not create false frequencies
\(k\)
frequency bin; bin \(k\) is at \(k \cdot 16{,}000/400 = 40k\) Hz, so there are 201 bins from 0 to 8,000 Hz
\(m\)
frame index (time); frame \(m\) starts at \(mH\)

Human hearing does not treat all frequencies equally. We hear the step from 100 Hz to 200 Hz as large, and the step from 7,000 Hz to 7,100 Hz as almost nothing. The mel scale models this: it is close to linear below 1 kHz and logarithmic above. Whisper (through librosa) uses the Slaney version [Certain]:

\[ \operatorname{mel}(f) = \begin{cases} \dfrac{3f}{200} & f \lt 1000 \text{ Hz} \\[2mm] 15 + 27\,\dfrac{\ln(f/1000)}{\ln 6.4} & f \ge 1000 \text{ Hz} \end{cases} \tag{184.2} \]

We place \(B\) triangular filters at equal distances on the mel scale between 0 and 8 kHz (so they are narrow at low frequencies and wide at high ones), sum the power under each triangle, take the logarithm, and normalize. In Whisper’s code this is:

\[ \begin{aligned} M[b, m] &= \log_{10} \max\!\Big(\textstyle\sum_k F[b, k]\, P[k, m],\; 10^{-10}\Big) \\ \tilde M &= \max\!\big(M,\; \max(M) - 8\big), \qquad L = \frac{\tilde M + 4}{4} \end{aligned} \tag{184.3} \]
Read the symbols
\(F[b, k]\)
weight of frequency bin \(k\) in mel filter \(b\) (a triangle, scaled so every filter has the same area)
\(B\)
number of mel filters: 80 in the original Whisper, 128 in Whisper large-v3 and in Qwen2-Audio and Qwen2.5-Omni [Certain]
\(\max(M) - 8\)
a floor 8 orders of magnitude (80 dB) below the loudest value, so silence does not become \(-10\)
\(L\)
the final input, roughly in \([-1, 1]\); the paper says the input is scaled to “between -1 and 1 with approximately zero mean” [Certain]

The result is a \(B \times T\) image: \(B\) mel bins tall and 100 columns per second wide. Whisper always pads or cuts audio to 30 seconds, so its input is \(128 \times 3{,}000\) (large-v3) [Certain]. We checked our filter bank against the one shipped with Whisper: the largest difference is about \(10^{-7}\).

Figure 2. One second of a synthetic sound at 16 kHz turned into a Whisper-style log-Mel spectrogram with (184.1) to (184.3): 400-sample Hann window, 160-sample hop, 100 frames. Top: the waveform. Middle: the spectrogram (bright = loud; the vertical axis is mel bins, low frequencies at the bottom). Right: the column of the selected frame.
How to read Figure 2

Start with the rising whistle: its frequency goes from 200 Hz to 4,000 Hz in one second, so at frame 50 (0.5 s) it is at 2,100 Hz. The readout shows the brightest mel bin and its center frequency (about 2,124 Hz with 128 mels, 2,042 Hz with 40 mels: fewer, wider filters mean a coarser answer). The line is straight at the bottom and bends at the top because the mel axis compresses high frequencies. Now pick the vowel: it is a stack of harmonics of 140 Hz, and the brightest band near 700 Hz is the first formant (a resonance of the mouth that tells “a” from “i”). The claps are short, wide-band bursts: vertical stripes. Speech is a mix of all three patterns, which is why one 2D image works so well.

Speech encoders: Whisper and its descendants

Whisper

Whisper (OpenAI, 2022) is an encoder-decoder Transformer trained on 680,000 hours of audio paired with transcripts found on the internet [Certain]. The labels are “weak” (not checked by hand), but there are so many of them that the model became robust to accents, noise and microphones. Five sizes were released, from Tiny (4 layers, width 384, 39M parameters) to Large (32 layers, width 1,280, 1,550M parameters) [Certain].

The encoder is short to describe [Certain]: a stem of two 1D convolutions over time (filter width 3, GELU), where the second has stride 2; then sinusoidal position embeddings; then pre-norm Transformer blocks; then a final LayerNorm. The mel bins are the input channels of the first convolution, so the convolution looks at 3 neighboring frames at a time across all frequencies. The stride halves the frame rate from 100 to 50 per second, so 30 s become 1,500 encoder frames of 20 ms each.

The decoder writes text with special tokens that set the task [Certain]: <|startoftranscript|>, then a language token (99 languages), then <|transcribe|> or <|translate|>, then either <|notimestamps|> or timestamp tokens. Timestamps are quantized to 20 ms, “the native time resolution of Whisper models”, which is exactly one encoder frame [Certain]. A <|nospeech|> token marks segments without speech. This is the same “special tokens as a program” idea as chat templates in session 9.

From Whisper to an LLM’s audio encoder

For an audio-language model, we keep only Whisper’s encoder and connect it to an LLM, like a vision encoder in session 177. Qwen2-Audio does this with Whisper large-v3 and adds an average pool with stride 2 after the Transformer, so each output vector covers about 40 ms [Certain]. Qwen2.5-Omni reuses that encoder (32 layers, width 1,280, 20 heads, 128 mels) and a linear projection to the LLM width of 3,584 [Certain]. In the released code, the number of tokens for \(T\) mel frames is:

\[ T_1 = \left\lfloor \frac{T - 1}{2} \right\rfloor + 1, \qquad T_{\text{audio}} = \left\lfloor \frac{T_1 - 2}{2} \right\rfloor + 1 \tag{184.4} \]
Read the symbols
\(T\)
number of mel frames (100 per second)
\(T_1\)
frames after the stride-2 convolution (50 per second, as in Whisper)
\(T_{\text{audio}}\)
tokens after the stride-2 average pool: 25 per second, one per 40 ms
Worked example: 30 seconds of speech

\(T = 3{,}000\) mel frames. After the convolution: \(\lfloor 2{,}999/2 \rfloor + 1 = 1{,}500\) (Whisper’s encoder length). After pooling: \(\lfloor 1{,}498/2 \rfloor + 1 = 750\) tokens. A 45 s clip gives \(4{,}500 \to 2{,}250 \to 1{,}125\) tokens. One minute gives 1,500 tokens. People speak roughly 150 words per minute [Likely], so that is about 10 audio tokens per spoken word, several times more than the same words cost as text tokens.

Two more changes make the encoder streamable in Qwen2.5-Omni [Certain]. The paper changes the encoder “from full attention over the entire audio to performing attention in blocks of 2 seconds each”; in the code this is n_window = 100, so each block is 200 mel frames. And the encoder is initialized from Whisper large-v3 and then trained with the LLM. From the config, the encoder has about 635M parameters (our count). With 2-second blocks, the encoder can process audio as it arrives, block by block, which is needed for chunked prefill in a live conversation.

Algorithm 1: Whisper-style audio front end and encoder (Qwen2.5-Omni variant)
  1. input: waveform at any rate; LLM width \(d_{\text{LLM}}\)
  2. resample to 16 kHz; for Whisper itself, pad or cut to 30 s
  3. STFT with a 400-sample Hann window and a 160-sample hop; keep \(|\cdot|^2\) // (184.1), 100 frames/s
  4. apply 128 mel filters, \(\log_{10}\), clamp to \(\max(M) - 8\), then \((x + 4)/4\) // (184.2), (184.3)
  5. split the frames into blocks of 200 (2 s) // streaming variant only
  6. conv1d (3, stride 1) \(\rightarrow\) GELU \(\rightarrow\) conv1d (3, stride 2) \(\rightarrow\) GELU; add sinusoidal positions // 50 frames/s
  7. 32 pre-norm Transformer layers, attention only inside each block
  8. average pool, size 2, stride 2; LayerNorm; linear to \(d_{\text{LLM}}\) // (184.4), 25 tokens/s
  9. wrap with audio-start and audio-end tokens; return the audio tokens

Audio tokens: continuous vectors or discrete codes

“Audio token” means two different things in practice, and interviews often test whether you know the difference.

  • Continuous audio tokens are the encoder output vectors above. They enter the LLM exactly like image tokens: through a projection, in place of placeholder tokens. They are good for understanding (speech recognition, audio questions), because the encoder keeps semantic and acoustic detail. But an LLM cannot generate them with a softmax, because they are not from a finite vocabulary.
  • Discrete audio tokens are integers from a codebook. There are two families. Semantic units come from clustering self-supervised speech features (for example k-means on HuBERT features [Certain]); they keep what was said but lose much of the voice. Acoustic codec tokens come from a neural audio codec such as SoundStream, EnCodec or Mimi: an encoder-quantizer-decoder trained to compress and rebuild the waveform. Codec tokens are what a model generates when it speaks.

Neural codecs quantize with residual vector quantization (RVQ). The first codebook approximates the vector; the second codebook approximates what is left (the residual); and so on:

\[ \mathbf{r}_0 = \mathbf{z}, \qquad q_m = \argmin_{j} \big\lVert \mathbf{r}_{m-1} - \mathbf{c}^{(m)}_j \big\rVert, \qquad \mathbf{r}_m = \mathbf{r}_{m-1} - \mathbf{c}^{(m)}_{q_m}, \qquad \hat{\mathbf{z}} = \sum_{m=1}^{N_q} \mathbf{c}^{(m)}_{q_m} \tag{184.5} \]
Read the symbols
\(\mathbf{z}\)
one latent vector from the codec encoder (one time step)
\(\mathbf{c}^{(m)}_j\)
entry \(j\) of codebook \(m\); each codebook has \(K\) entries
\(q_m\)
the integer token chosen in codebook \(m\)
\(\mathbf{r}_m\)
the residual left after \(m\) codebooks; it shrinks as \(m\) grows
\(N_q\)
number of codebooks used; fewer codebooks = lower bitrate and lower quality

Each time step produces \(N_q\) integers of \(\log_2 K\) bits, so the bitrate and the number of tokens per second are:

\[ \text{bitrate} = f_{\text{frame}} \cdot N_q \cdot \log_2 K, \qquad \text{tokens per second} = f_{\text{frame}} \cdot N_q \tag{184.6} \]

EnCodec at 24 kHz has 75 frames per second (encoder strides 2, 4, 5, 8, so \(24{,}000/320 = 75\)) and up to 32 codebooks of 1,024 entries (10 bits) [Certain]. With 8 codebooks: \(75 \times 8 \times 10 = 6{,}000\) bits/s = 6 kbps and 600 tokens per second; with 2 codebooks, 1.5 kbps [Certain]. Kyutai’s Mimi codec (used by the Moshi dialogue model) runs at only 12.5 frames per second with 8 RVQ levels [Certain], which is 100 tokens per second and, with 2,048-entry codebooks, about 1.1 kbps [Likely]. Mimi also distills semantic information from a self-supervised speech model into its first level [Certain], so one codec mixes both token families.

Our code below runs RVQ with k-means codebooks of 64 entries on random 8-dimensional vectors. The relative error after 1, 2, 3, 4 codebooks is 0.645, 0.417, 0.272, 0.179: each extra codebook removes about a third of what is left.

Figure 3 compares the token cost of one clip in each representation, next to the video tokens of the same clip. For Qwen2.5-Omni the total for a video with sound is:

\[ n_{\text{tokens}} \approx \underbrace{\left\lfloor \frac{S f}{2} \right\rfloor \cdot \frac{\hat H}{28} \cdot \frac{\hat W}{28}}_{\text{video: frame pairs} \times \text{tokens per pair}} \;+\; \underbrace{25\, S}_{\text{audio}} \tag{184.7} \]
Read the symbols
\(S\)
clip length in seconds
\(f\)
video sampling rate in frames per second; two frames form one temporal patch, as in Qwen2.5-VL (session 182)
\(\hat H, \hat W\)
frame size after rounding to multiples of 28 (patch 14 × merge 2); a 640×360 frame becomes 644×364, so \(13 \times 23 = 299\) tokens per frame pair
Figure 3. Tokens for one clip. Top bar: video tokens in Qwen2.5-Omni (184.7). The other bars are the audio of the same clip in four representations: Qwen2.5-Omni encoder output (25/s), Whisper encoder frames (50/s), Mimi codes (12.5 frames × 8 levels), and EnCodec at 6 kbps (75 frames × 8 codebooks). The scale is logarithmic.
How to read Figure 3

At the default (60 s, 2 fps, 640×360), video costs 17,940 tokens and Omni audio 1,500: sound is only 7.7% of the 19,440 tokens. Switch to 1280×720 and the video jumps to 71,760 while the audio stays at 1,500. Now compare the audio bars: if an LLM had to read EnCodec codes it would need 36,000 tokens for one minute, 24 times the encoder vectors. That is why understanding uses continuous encoder tokens, and why speech generators use low-rate codecs (Mimi at 6,000 codes per minute) or predict several codebooks per step.

Aligning audio and video in time

When a video has sound, the model must know that the bang it hears at 4.0 s belongs to the frame at 4.0 s. There are two separate problems. Order: where do audio tokens go in the 1D sequence? Position: what position ids do they get, so that attention knows which audio and video tokens are close in time?

Positions: TMRoPE

Qwen2-VL introduced M-RoPE: each token has three position ids \((t, h, w)\) for time, row and column, and the rotary dimensions are split into three groups, one per id (session 14). In the 7B Thinker the 64 frequency pairs of each head are split as 16 for time, 24 for height and 24 for width [Certain]. Qwen2.5-Omni’s TMRoPE (time-aligned multimodal RoPE) fixes the meaning of the time id: one temporal id = 40 ms of real time, for audio and video alike [Certain] (position_id_per_seconds = 25 in the config). With a start id \(s_0\):

\[ \begin{aligned} \text{audio token } j: &\quad (t, h, w) = (s_0 + j,\; s_0 + j,\; s_0 + j) \\ \text{video token at row } i, \text{ col } j \text{ of pair } g: &\quad (t, h, w) = \Big(s_0 + \Big\lfloor 25 \cdot g \cdot \frac{2}{f} \Big\rfloor,\; s_0 + i,\; s_0 + j\Big) \end{aligned} \tag{184.8} \]
Read the symbols
\(s_0\)
the first id of this video (one more than the largest id before it)
\(j\) (audio)
index of the audio token; token \(j\) covers \([0.04j,\, 0.04(j+1))\) seconds
\(g\)
index of the frame pair (temporal patch); it starts at \(g \cdot 2/f\) seconds
\(f\)
video frames per second; at \(f = 2\), consecutive pairs are 1 s apart, so their time ids differ by 25
\(i, j\) (video)
row and column of the merged 2×2 patch inside the frame

Audio gets the same id on all three axes, so for audio-to-audio attention TMRoPE is ordinary 1D RoPE. Video and audio that happen at the same moment get the same time id, so RoPE sees them as close, whatever their order in the sequence. Text after the video continues from the largest id + 1 [Certain].

Order: time interleaving in 2-second chunks

If we put all video tokens first and all audio last, the first audio token is thousands of positions away from the frame it belongs to, and a streaming model would have to wait for the whole video. Qwen2.5-Omni instead cuts the video-with-audio into chunks of 2 seconds of real time, and in each chunk places the video tokens first, then the audio tokens [Certain]. In the code a token belongs to chunk \(c\) when its time id satisfies

\[ 50\,c \;\le\; t - s_0 \;\lt\; 50\,(c + 1), \qquad 50 = 25 \text{ ids/s} \times 2 \text{ s} \tag{184.9} \]
video tokenaudio token (40 ms)selected token and its time partners
Figure 4. TMRoPE ids (184.8) and time interleaving (184.9) for a short video with sound. Each row is one 2-second chunk in sequence order: its video tokens, then its audio tokens. The frames are tiny (2×2 or 3×3 merged tokens) so that every token fits on screen; a real 640×360 frame pair has 299. Click any token.
How to read Figure 4

The default (6 s, 2 fps, 2×2) gives 174 tokens in the order V8 A50 V8 A50 V8 A50: two frame pairs (8 video tokens) and 50 audio tokens per chunk. Click the first video token of the second row: its time id is 50 (2.0 s), and the audio token at 2.0 s, 8 positions later, has the same time id. Switch to 4 fps: four pairs per chunk (V16), time ids 0, 12, 25, 37, … (the floor in (184.8)). Switch to 0.5 fps with 8 s: some chunks have no frame at all, and the audio simply continues. In every case the text after the video starts at id 150 for 6 s, which is “largest id + 1”.

Algorithm 2: TMRoPE position ids with time interleaving (video with audio)
  1. input: video tokens grouped by frame pair \(g\) with merged grid \(h \times w\); audio tokens \(0 \dots A-1\); fps \(f\); start id \(s_0\)
  2. for each video token: \(t \leftarrow s_0 + \lfloor 25 g \cdot 2/f \rfloor\), \(h \leftarrow s_0 + i\), \(w \leftarrow s_0 + j\) // (184.8)
  3. for each audio token \(j\): \(t = h = w \leftarrow s_0 + j\)
  4. for chunk \(c = 0, 1, \dots\): append the video tokens with \(t - s_0 \in [50c, 50c + 50)\), then the audio tokens in the same range // (184.9)
  5. next id for text \(\leftarrow 1 + \max(\text{ids used})\)
  6. apply RoPE with the frequency pairs split (16, 24, 24) among \(t, h, w\)
  7. return the reordered tokens and their \((t, h, w)\) ids

Other systems solve the same problem differently. Many video-LLMs drop audio entirely, or run a separate speech recognizer and insert the transcript with timestamps as text (simple, but it loses tone, music and sound events). Some interleave per frame instead of per chunk. The trade-off is the same as in session 179: interleaving in one sequence gives full attention between modalities at the cost of a long sequence.

Omni models: text and speech out

The cascade and why it is not enough

The classic voice assistant is a cascade: a speech recognizer (ASR) writes the user’s words as text, an LLM answers in text, and a text-to-speech system (TTS) reads the answer aloud. It is modular and easy to debug. But it has two problems. It loses everything that is not words: tone, emotion, laughter, background sound. And it is slow: each stage waits for the previous one. Moshi’s authors describe the latency of such pipelines as “several seconds”, while Moshi itself reaches 160 ms in theory and 200 ms in practice [Certain].

Thinker and Talker

Qwen2.5-Omni keeps the strong text LLM and adds a mouth [Certain]. The Thinker (initialized from Qwen2.5-7B: 28 layers, width 3,584) generates text as usual. The Talker is a separate decoder (24 layers, width 896, 12 heads, from the config) with a vocabulary of 8,448 tokens that include the speech codes of a new codec called qwen-tts-tokenizer [Certain]. The Talker is “dual-track”, inspired by Mini-Omni: at each step it receives both the Thinker’s high-level hidden states and the embeddings of the text tokens the Thinker sampled, and shares the Thinker’s full context [Certain].

Why both? The paper gives two reasons [Certain]. Hidden states carry tone and attitude before the whole sentence is written, which a streaming voice needs. But hidden states encode meaning, not sound: two words with similar meaning but different pronunciation can have similar hidden states, so the sampled text tokens are needed to remove that ambiguity. The speech does not need word-level timing alignment with the text, which simplifies training data [Certain].

From codes to waveform, in a stream

The token2wav part has two models [Certain]: a DiT (diffusion Transformer, 22 layers, width 1,024) trained with flow matching turns codes into an 80-bin mel spectrogram, and a modified BigVGAN vocoder turns the mel spectrogram into a waveform (see session 229 and session 228). From the config, each code is repeated twice before the DiT, and BigVGAN upsamples each mel frame by \(5 \cdot 3 \cdot 2 \cdot 2 \cdot 2 \cdot 2 = 240\) samples, so one code is 480 samples; at a 24 kHz output rate that is 50 codes per second [Likely].

To stream, the DiT must not look at the whole future. Codes are grouped into blocks, and each block attends only to a window: 2 blocks back, itself, and 1 block ahead, a receptive field of 4 blocks [Certain]:

\[ A_{uv} = \begin{cases} 1 & \text{if } -2 \le b(v) - b(u) \le 1 \\ 0 & \text{otherwise} \end{cases} \tag{184.10} \]
Read the symbols
\(u, v\)
positions in the DiT input (query and key)
\(b(\cdot)\)
the block a position belongs to
\(A_{uv}\)
1 if \(u\) may attend to \(v\)

So the waveform of block \(b\) can be produced as soon as block \(b+1\) of codes exists: the delay is one block, not the whole answer. Together with 2-second blocks in the audio encoder (chunked prefill) and a streaming Talker, this is how the first audio packet arrives quickly. The paper lists four latency sources: input processing, time from first text to first speech token, turning the first speech segment into audio, and the model’s own size and FLOPs [Certain].

Training and results

Pretraining has three stages [Certain]: (1) freeze the LLM and train the vision and audio encoders (adapters first); (2) unfreeze everything and add 800B tokens of image and video data, 300B of audio data and 100B of video-with-audio data, with sequences up to 8,192; (3) extend to 32k tokens with long audio and video. The Talker is trained in three more stages: in-context speech continuation, then DPO with rewards based on word error rate and pause errors to make speech generation stable, then speaker fine-tuning for specific voices [Certain]. On LibriSpeech, Qwen2.5-Omni-7B reports a word error rate of 1.8% (test-clean) and 3.4% (test-other), next to 1.8% and 3.6% for Whisper large-v3 in the same table [Certain]. So adding vision and speech output did not cost it speech recognition quality.

DesignExampleSpeech inSpeech out
CascadeASR → LLM → TTStext transcript onlyseparate TTS after the text
Audio encoder + LLMQwen2-Audiocontinuous encoder tokensnone (text only)
Thinker–TalkerQwen2.5-Omni, Mini-Omni (text-guided speech)continuous encoder tokenscodec tokens from a second decoder, streamed
Full duplex, parallel streamsMoshi (Mimi codec)codec tokens of the user’s streamcodec tokens of its own stream, with “inner monologue” text first

Applications: speakers, sound events and commentary

Active speaker detection

Active speaker detection (ASD) answers, for every face in every frame, “is this person speaking now?”. It is needed for meeting transcripts with names, for auto-framing in video calls, for dubbing, and for “who said X?” questions in Video-RAG. The naive rule “the lips move, so the person speaks” fails: people chew, smile and mouth along, and in dubbed films the lips do not match the voice. The reliable signal is synchrony: the mouth opens when the voice gets louder, at the same moment.

A simple synchrony score correlates the audio envelope \(a\) with the mouth-opening signal \(m_k\) of face \(k\) in a window around frame \(t\), after shifting the audio by a lag \(\delta\):

\[ s_k(t, \delta) = \operatorname{corr}\big(a_{t' - \delta},\, m_{k, t'}\big)_{t' \in [t - W/2,\, t + W/2]}, \qquad \hat\delta = \argmax_{\delta} \sum_k \max\big(0,\, \operatorname{corr}(a_{\cdot - \delta}, m_{k,\cdot})\big) \tag{184.11} \]
Read the symbols
\(a_t\)
loudness of the audio at video frame \(t\) (25 frames per second, 40 ms each)
\(m_{k,t}\)
how open the mouth of face \(k\) is at frame \(t\)
\(W\)
window length in frames; longer windows give more evidence but blur speaker changes
\(\delta\)
audio-video offset in frames; a whole file is often a few frames out of sync
\(\hat\delta\)
the offset that best explains all faces, estimated once per clip

TalkNet (2021) learns this instead of hand-coding it [Certain]. A visual temporal encoder reads 112×112 face crops (receptive field 21 frames, 840 ms at 25 fps); an audio temporal encoder reads 13-dimensional MFCC features (receptive field 189 audio frames, 1,890 ms). Then cross-attention in both directions lets each stream query the other:

\[ F_{a \to v} = \softmax\!\left(\frac{Q_v K_a^\top}{\sqrt{d}}\right) V_a, \qquad F_{v \to a} = \softmax\!\left(\frac{Q_a K_v^\top}{\sqrt{d}}\right) V_v \tag{184.12} \]

Here the queries come from one modality and the keys and values from the other (see session 20). Because attention can look a few frames to either side, it absorbs small offsets that a fixed frame-by-frame comparison cannot. A self-attention layer over the joined features then models the whole utterance, and a classifier labels each frame. TalkNet reached 92.3% mAP on the AVA-ActiveSpeaker validation set and 90.8% on its test set [Certain]. Its key ablation is about time: with a fixed input of 5 frames (0.2 s) it gets 75.2% mAP, with 25 frames (1 s) 87.9%, with 50 frames (2 s) 89.5% [Certain]. Removing the whole attention part drops it to 90.0% [Certain]. Figure 5 lets you feel both effects on a toy scene.

audio loudnessmouth openingpredicted speaking, correctpredicted wrong
Figure 5. A toy active speaker detector on 10 seconds at 25 fps (seeded random data). Face A and face B take turns speaking; their mouths follow the syllables of the audio. Face C is chewing: its mouth moves all the time, but not with the voice. Each face row shows the mouth signal and, below it, the per-frame prediction colored by correctness. A frame is “speaking” when the audio in the window is loud and the rule fires: mouth variance above a threshold, or correlation (184.11) above 0.3.
How to read Figure 5

Start with “mouth moves”: A and B are fine, but C is labeled as speaking whenever anyone talks, so C’s accuracy is 6.0% and the mean is 63.2%. Switch to sync: C drops out (95.6%) and the mean is 90.1%. Now move the window: 5 frames (0.2 s) is too little evidence for C (74.0%), 51 frames blurs the turn changes for A (80.8%); this is TalkNet’s long-context finding in miniature. Then set the audio delay to 3 frames (120 ms): sync breaks (mean 69.1%) because the mouth now leads the sound. Tick “estimate and correct the offset”: the lag search of (184.11) finds 3 and accuracy returns to 90.1%.

Algorithm 3: active speaker detection for a video clip
  1. input: video at 25 fps with audio; face detector and tracker
  2. detect and track faces; crop each face track to a fixed size (for example \(112 \times 112\), grayscale)
  3. compute audio features at a rate that is a multiple of the video rate (MFCC or log-Mel at 100 frames/s: \(100 / 25 = 4\) per video frame)
  4. estimate one global offset \(\hat\delta\) by maximizing audio–mouth agreement over the clip; shift the audio // (184.11)
  5. for each face track: encode the visual stream and the audio stream over a long window (1 to 2 s or the whole track)
  6. fuse with cross-attention in both directions, then self-attention over time // (184.12)
  7. classify each frame as speaking / not speaking; smooth the labels (for example a median filter over a few frames)
  8. return per-frame, per-face speaking scores; join with the ASR transcript by time to name the speaker of each sentence

Audio events and commentary understanding

Audio events are sounds that are not speech: a goal horn, glass breaking, a siren, applause, music. The standard dataset is Google’s AudioSet, with about 2 million 10-second YouTube clips labeled with 527 classes [Likely]. In video systems, audio events are cheap and strong triggers: a sudden crowd roar or a whistle often marks a key moment before anything clear happens on screen, and a gunshot or scream is a strong moderation signal. An omni model can do this zero-shot (“list the sounds and their times”), and with TMRoPE it can answer with times in seconds.

Commentary understanding combines three audio signals. The words of the commentator (“what a save!”) name the event; the prosody (pitch and loudness rising) tells how important it is; and the crowd noise confirms it. A cascade with ASR keeps only the words. A sports highlight system built on an omni model, or on an audio encoder plus a video encoder, can use all three. A practical recipe: compute a loudness and pitch track for the whole match (very cheap), run the omni model only on windows around the peaks, and ask it to describe the event and give its start time.

Code: the key mechanisms in one file

This script builds the Whisper-style log-Mel front end (184.1 to 184.3), a tiny audio encoder with the same shape rules as Qwen2.5-Omni’s (184.4), the TMRoPE ids with time interleaving (184.8, 184.9), and residual vector quantization (184.5). It runs on CPU in under a second. Our mel filter bank matches the one shipped with Whisper to about \(10^{-7}\).

import math, torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)
SR, N_FFT, HOP = 16000, 400, 160              # 16 kHz, 25 ms window, 10 ms hop

def mel(f):                                    # Slaney mel scale (184.2), as in librosa / Whisper
    return 3 * f / 200 if f < 1000 else 15 + 27 * math.log(f / 1000) / math.log(6.4)
def hz(m):
    return 200 * m / 3 if m < 15 else 1000 * math.exp((m - 15) * math.log(6.4) / 27)

def mel_filters(n_mels, sr=SR, n_fft=N_FFT):  # triangles, equally spaced in mel
    pts = torch.tensor([hz(m) for m in torch.linspace(0, mel(sr / 2), n_mels + 2).tolist()])
    f = torch.linspace(0, sr / 2, n_fft // 2 + 1)
    lo, ce, hi = pts[:-2, None], pts[1:-1, None], pts[2:, None]
    tri = torch.clamp(torch.minimum((f - lo) / (ce - lo), (hi - f) / (hi - ce)), min=0)
    return tri * 2 / (hi - lo)                 # equal area per filter

def log_mel(x, n_mels=128):                    # Whisper-style log-Mel (184.1)-(184.3)
    S = torch.stft(x, N_FFT, HOP, window=torch.hann_window(N_FFT), return_complex=True)
    S = S[..., :-1].abs() ** 2                 # drop the last frame: 100 frames per second
    m = torch.clamp(mel_filters(n_mels) @ S, min=1e-10).log10()
    m = torch.maximum(m, m.max() - 8.0)        # keep an 80 dB range
    return (m + 4.0) / 4.0

class AudioEncoder(nn.Module):                 # conv stem + transformer + pool + adapter
    def __init__(s, n_mels=128, d=64, d_llm=96, layers=2):
        super().__init__()
        s.conv1 = nn.Conv1d(n_mels, d, 3, padding=1)
        s.conv2 = nn.Conv1d(d, d, 3, stride=2, padding=1)       # 100 -> 50 frames/s
        layer = nn.TransformerEncoderLayer(d, 4, 4 * d, batch_first=True, norm_first=True)
        s.blocks = nn.TransformerEncoder(layer, layers, enable_nested_tensor=False)
        s.pool = nn.AvgPool1d(2, stride=2)                      # 50 -> 25 tokens/s
        s.proj = nn.Linear(d, d_llm)                            # to the LLM width
    def forward(s, mel):                       # mel: (B, n_mels, T)
        h = F.gelu(s.conv2(F.gelu(s.conv1(mel)))).transpose(1, 2)
        T, d = h.shape[1], h.shape[2]
        pos = torch.arange(T)[:, None] / 10000 ** (torch.arange(0, d, 2) / d)
        h = h + torch.cat([pos.sin(), pos.cos()], dim=1)        # sinusoidal positions
        h = s.pool(s.blocks(h).transpose(1, 2)).transpose(1, 2)
        return s.proj(h)

t = torch.arange(30 * SR) / SR                 # 30 s: a 220 Hz tone plus a little noise
x = 0.5 * torch.sin(2 * math.pi * 220 * t) + 0.01 * torch.randn(t.shape)
M = log_mel(x)
tok = AudioEncoder()(M[None])
print(tuple(M.shape), tuple(tok.shape), tok.shape[1] / 30)

def tmrope_ids(seconds, fps=2, H=2, W=2, rate=25, chunk=2, tp=2, st=0):
    """(kind, t, h, w) for video-with-audio, time-interleaved in 2 s chunks (184.8), (184.9)."""
    n_grid = int(seconds * fps / tp)                            # one grid = tp frames
    vid = [("V", st + int(g * tp / fps * rate), st + i, st + j)
           for g in range(n_grid) for i in range(H) for j in range(W)]
    aud = [("A", st + k, st + k, st + k) for k in range(int(seconds * rate))]
    per, seq = chunk * rate, []
    for c in range(math.ceil(seconds / chunk)):
        inside = lambda p: c * per <= p[1] - st < (c + 1) * per
        seq += [p for p in vid if inside(p)] + [p for p in aud if inside(p)]
    return seq

seq = tmrope_ids(6)
runs = [[seq[0][0], 0]]
for p in seq:
    if p[0] == runs[-1][0]: runs[-1][1] += 1
    else: runs.append([p[0], 1])
print(len(seq), " ".join(f"{k}{n}" for k, n in runs))
at4 = [i for i, p in enumerate(seq) if p[1] == 100]             # temporal id 100 = 4.0 s
print(at4, seq[at4[0]], seq[at4[-1]])
print("next text id:", max(max(p[1:]) for p in seq) + 1)

def rvq(z, n_books=4, K=64, iters=10):         # residual vector quantization (184.5)
    r, out = z.clone(), []
    for m in range(n_books):
        C = r[torch.randperm(len(r))[:K]].clone()               # k-means codebook on the residual
        for _ in range(iters):
            idx = torch.cdist(r, C).argmin(1)
            for k in range(K):
                if (idx == k).any(): C[k] = r[idx == k].mean(0)
        idx = torch.cdist(r, C).argmin(1)
        r = r - C[idx]
        out.append(round((r.norm() / z.norm()).item(), 3))
    return out

print(rvq(torch.randn(4096, 8)))               # relative error after 1, 2, 3, 4 codebooks

# Output:
# (128, 3000) (1, 750, 96) 25.0
# 174 V8 A50 V8 A50 V8 A50
# [116, 117, 118, 119, 124] ('V', 100, 0, 0) ('A', 100, 100, 100)
# next text id: 150
# [0.645, 0.417, 0.272, 0.179]

Three things to notice in the output. Thirty seconds become a \(128 \times 3{,}000\) spectrogram and then 750 tokens, 25 per second. In the interleaved sequence, the video tokens at 4.0 s (positions 116 to 119) and the audio token at 4.0 s (position 124) are 5 to 8 positions apart and share time id 100. And the text after a 6 s video starts at id 150. To run the real model, use Qwen2_5OmniForConditionalGeneration and Qwen2_5OmniProcessor in Hugging Face Transformers, with use_audio_in_video=True to interleave the soundtrack [Certain].

Where this is used

TaskAudio partWhy audio helps
Subtitles, meeting notes, lecture searchWhisper-style ASR with 20 ms timestampsThe words are in the sound, not in the frames
“Who said what” in meetings, interviews, filmsActive speaker detection (Figure 5) + ASR + face tracksJoins voice to face by synchrony, not by lip motion alone
Key-moment detection, sports highlightsLoudness and crowd-noise peaks, commentator prosody, an omni model on the peak windowsSound often marks the moment before the picture makes it clear
Video retrieval and Video-RAGTranscript and audio-event tags with timestamps, indexed next to frame embeddingsMany queries are about what was said (“the part where she explains the API”)
Video classification and moderationAudio-event classifier (gunshots, screams, music genre)Cheap and robust to bad lighting or off-screen events
Video fingerprinting and near-duplicate detectionSpectrogram-based audio fingerprintsRe-uploads are often cropped or re-colored but keep the soundtrack
Voice assistants and live video chatOmni model with streaming encoder and TalkerLow latency, and tone of voice is kept in both directions

Common mistakes

  • Wrong sample rate. Feeding 44.1 kHz or 48 kHz audio to a 16 kHz model without resampling shifts every frequency, and the model hears a different voice. Always resample to the processor’s sampling_rate.
  • Mismatched mel settings. Whisper large-v3 and Omni use 128 mel bins; older Whisper models use 80. Features from the wrong extractor give garbage without any error message.
  • Counting audio tokens with the wrong rate. Whisper’s encoder emits 50 frames per second; Qwen2-Audio and Qwen2.5-Omni emit 25 tokens per second after pooling; a codec at 75 Hz with 8 codebooks is 600 tokens per second if flattened. Use (184.4) and (184.6), not a guess.
  • Dropping the soundtrack of a video. Many pipelines decode only the frames. For Qwen2.5-Omni you must pass use_audio_in_video=True; otherwise the model answers from pixels alone.
  • Ignoring audio–video offset. Containers and re-encoding often shift audio by tens to hundreds of milliseconds. Sync-based methods break (Figure 5). Estimate the offset per clip, or use a model with enough temporal context to absorb it.
  • Using lip motion as the speaking signal. Chewing, smiling and dubbed speech all move the lips. Use audio–visual synchrony and long windows.
  • Treating continuous audio embeddings as generatable. An LLM can only sample from a finite vocabulary; to produce speech you need codec tokens and a decoder (or a separate TTS).
  • Measuring only text quality of an omni model. Latency to the first audio packet, speech naturalness and voice stability are separate metrics; a model can be excellent in text and still sound robotic or slow.

Interview questions

How does Whisper turn a waveform into encoder input, and how many encoder frames is 30 seconds?basic
Resample to 16 kHz, pad or cut to 30 s, compute an STFT with a 25 ms (400-sample) Hann window and a 10 ms (160-sample) hop, apply 80 (or 128 in large-v3) mel filters, take \(\log_{10}\), clamp to 8 orders below the maximum and scale with \((x+4)/4\). That gives 3,000 frames. The conv stem (two width-3 convolutions, the second with stride 2) halves this to 1,500 frames of 20 ms, which is also the resolution of Whisper’s timestamp tokens.
Why use a log-Mel spectrogram instead of the raw waveform?basic
The waveform is long (480,000 samples for 30 s) and changes a lot with phase, pitch and microphone, while what matters for speech is which frequencies carry energy over time. The STFT exposes that; the mel scale spends resolution where hearing is sensitive (low frequencies); the log compresses the huge dynamic range so quiet and loud sounds are both visible. The result is a compact 2D image (80 or 128 × 100 per second) that convolutions and Transformers handle well. Learned front ends on raw waveforms exist (wav2vec 2.0 style) but need more data.
How many LLM tokens does one minute of audio cost in Qwen2.5-Omni, and where does that number come from?core
1,500 tokens. 60 s give 6,000 mel frames; the stride-2 convolution gives \(\lfloor 5{,}999/2 \rfloor + 1 = 3{,}000\) frames; the stride-2 average pool after the Transformer gives \(\lfloor 2{,}998/2 \rfloor + 1 = 1{,}500\). So 25 tokens per second, 40 ms each, plus audio-start and audio-end tokens. For comparison, one minute of 640×360 video at 2 fps is 17,940 tokens.
Continuous audio embeddings vs discrete codec tokens: when do you use which?core
For understanding (ASR, audio QA, captioning sound), use continuous encoder outputs: they are rich and short (25 per second). For generating speech, the model must sample from a vocabulary, so it predicts discrete tokens from a neural codec (RVQ codes) or semantic units, and a decoder turns them back into sound. Many systems use both: continuous in, discrete out (Qwen2.5-Omni). Fully discrete systems like Moshi model the user’s audio as codec tokens too, which allows full-duplex speech-to-speech, at a higher token rate.
Explain residual vector quantization and compute EnCodec’s bitrate with 8 codebooks.core
RVQ quantizes a vector with codebook 1, then quantizes the remaining error with codebook 2, and so on; the reconstruction is the sum of the chosen entries (184.5). Each extra codebook adds detail, so one model can serve several bitrates by using more or fewer codebooks. EnCodec at 24 kHz has 75 frames per second and 1,024-entry codebooks (10 bits): \(75 \times 8 \times 10 = 6{,}000\) bit/s = 6 kbps, and 600 tokens per second. The downside for LLMs is the token count; solutions are low frame rates (Mimi at 12.5 Hz) or predicting all codebooks of a step at once.
What is TMRoPE, and why does Qwen2.5-Omni interleave audio and video in 2-second chunks?advanced
TMRoPE is M-RoPE (separate rotary parts for time, height, width) with absolute time: one temporal id is 40 ms for both audio and video. Audio tokens get the same id on all three axes; video tokens get \(t = \lfloor 25 g \cdot 2/f \rfloor\) for frame pair \(g\), so the id is independent of the frame rate, and a sound and a frame at the same moment share a time id. Interleaving in 2 s chunks (video first, then audio) keeps related tokens close in the sequence, which helps attention and makes streaming and chunked prefill possible: the model can process chunk \(c\) before chunk \(c+1\) arrives, which matches the 2 s attention blocks of the audio encoder.
In the Thinker–Talker design, why does the Talker read the Thinker’s hidden states and not only the generated text?advanced
A streaming voice must choose tone and attitude before the sentence is finished; the Thinker’s hidden states carry this information about where the answer is going, and about the conversation (for example that the user sounded upset). Text tokens alone would be a cascade TTS that loses this. But hidden states are semantic, so words with similar meaning and different sounds can look alike; the sampled text tokens remove that ambiguity. Feeding both lets the Talker stream speech without word-level alignment, and the whole system trains end to end.
Where does latency come from in a spoken assistant, and how does an omni model reduce it?advanced
Sources: waiting for and encoding the input, time from the first text token to the first speech token, turning the first speech tokens into audio, and model size and FLOPs. A cascade adds them serially: ASR must finish, then the LLM, then TTS. Qwen2.5-Omni encodes audio in 2 s blocks (chunked prefill), lets the Talker start from the Thinker’s first hidden states, and decodes codes to waveform with a DiT whose attention sees only 2 blocks back and 1 ahead (184.10), plus chunked BigVGAN. Moshi goes further with full duplex: it listens and speaks at the same time in parallel streams, reaching about 200 ms in practice.
Design an active speaker detector for meeting videos. What features, what context, and what can go wrong?core
Track faces, crop mouths or faces (112×112), compute log-Mel or MFCC audio at 4 frames per video frame. Use synchrony rather than lip motion: encode each stream with temporal convolutions over 1 to 2 s, fuse with cross-attention both ways (TalkNet), then self-attention over the track. Context matters: TalkNet goes from 75.2% mAP with 0.2 s inputs to 89.5% with 2 s. Pitfalls: audio offset (estimate and correct it), off-screen speakers (audio is loud but no face is in sync; output “nobody visible”), overlapping speech, profile faces and occluded mouths, and dubbed content.
Cascade (ASR → LLM → TTS) or end-to-end omni model for a product? Give the trade-offs.core
Cascade: modular, each part replaceable and testable, the LLM can be any text model, transcripts are logged for free; but latency adds up to seconds, paralinguistic cues (emotion, laughter, background) are lost, and turn-taking is rigid. End-to-end: lower latency, keeps tone, can react to sounds and interruptions; but harder to train and evaluate, fewer model choices, voice safety (impersonation) and consistency need extra work. A common middle ground: an audio-encoder LLM for input (keeps tone) and a streaming TTS for output.
You must find highlight moments in 90-minute football matches. How do you use audio, and what do you send to an omni model?advanced
90 minutes at 2 fps and 640×360 would be about 1.6M video tokens plus 135,000 audio tokens: far too many for one call. First pass, cheap signal processing: loudness of the crowd and the commentator, pitch rise of the commentator, whistle detection with a small audio-event model. Pick the top peaks. Second pass: for each peak, send a 20 to 30 s window with video and audio interleaved (TMRoPE gives correct times) to the omni model and ask for the event type and its start second. Evaluate with recall of annotated events within a tolerance (for example ±5 s) and the cost per match.

Exercises

  1. How many Whisper encoder frames and how many Qwen2.5-Omni audio tokens does a 45-second clip produce? Use (184.4).
  2. A 10-second video at 1 fps with 640×360 frames and its soundtrack goes into Qwen2.5-Omni. How many video tokens, how many audio tokens, and what is the time-id step between consecutive frame pairs?
  3. EnCodec at 24 kHz with 4 codebooks: what is the bitrate, and how many codes does 10 s of audio produce?
  4. At 2 fps, in which 2-second chunk is the frame pair that starts at 3.0 s, and what is its time id (with \(s_0 = 0\))? Which audio token has the same time id?
  5. In Figure 5, the audio is 120 ms late. How many video frames is that at 25 fps, and why does a frame-by-frame correlation fail while TalkNet’s cross-attention can still work?
  6. Compare one minute of speech as Mimi codes and as EnCodec 6 kbps codes. How many tokens each, and what is the ratio?
Answerscheck
1. \(T = 4{,}500\) mel frames; \(T_1 = \lfloor 4{,}499/2 \rfloor + 1 = 2{,}250\) encoder frames (Whisper rate, 50/s); \(T_{\text{audio}} = \lfloor 2{,}248/2 \rfloor + 1 = 1{,}125\) tokens. 2. 10 frames = 5 frame pairs × 299 = 1,495 video tokens; audio \(25 \times 10 = 250\); total 1,745. Each pair covers 2 s, so the time id grows by \(25 \times 2 = 50\) per pair. 3. \(75 \times 4 \times 10 = 3{,}000\) bit/s = 3 kbps; \(75 \times 4 \times 10\text{ s} = 3{,}000\) codes. 4. Pair \(g = 3\) (pairs start at 0, 1, 2, 3 s), time id \(\lfloor 25 \cdot 3 \cdot 1 \rfloor = 75\); \(50 \le 75 \lt 100\), so chunk 1 (2 to 4 s). Audio token 75 (covering 3.00 to 3.04 s) has the same id. 5. \(0.120 / 0.040 = 3\) frames. The mouth now opens 3 frames before the sound, so within each window the two signals are shifted and their correlation drops (the mean accuracy in Figure 5 falls from 90.1% to 69.1%). Attention compares each visual frame with all audio frames in its context, so it can match the mouth to the sound 3 frames later; learning with naturally offset training data teaches it to do so. 6. Mimi: \(12.5 \times 8 \times 60 = 6{,}000\) tokens; EnCodec 6 kbps: \(75 \times 8 \times 60 = 36{,}000\) tokens; EnCodec needs 6 times more.

Key takeaways

  • Audio enters models as a log-Mel spectrogram: 16 kHz, 25 ms windows, 10 ms hop, 80 or 128 mel bins, log and normalize (184.1 to 184.3). It is a \(B \times 100\)-per-second image.
  • Whisper’s encoder (conv stem with stride 2, Transformer) gives 50 frames per second; LLM audio encoders (Qwen2-Audio, Qwen2.5-Omni) pool to 25 tokens per second (184.4). One minute = 1,500 tokens, far less than video.
  • Continuous encoder tokens are for understanding; discrete codec tokens (RVQ, 184.5, 184.6) are for generating speech. Codec token rates vary a lot: 600/s for EnCodec 6 kbps, 100/s for Mimi.
  • TMRoPE gives audio and video one shared clock (one id = 40 ms, 184.8) and interleaves them in 2-second chunks, video first (184.9).
  • Thinker–Talker: a text LLM thinks, a second decoder speaks from its hidden states and text, and a block-windowed DiT (184.10) plus a vocoder stream the waveform.
  • Audio–visual tasks need synchrony and long context: active speaker detection uses audio–mouth agreement (184.11) and cross-attention (184.12), not lip motion alone.

Key formulas

\[ L = \frac{\max\big(M, \max(M) - 8\big) + 4}{4}, \quad M = \log_{10}\!\big(F\, |\operatorname{STFT}(x)|^2\big), \qquad T_{\text{audio}} = \left\lfloor \frac{\lfloor (T-1)/2 \rfloor - 1}{2} \right\rfloor + 1 \]
\[ \text{bitrate} = f_{\text{frame}}\, N_q \log_2 K, \qquad t_{\text{video}} = s_0 + \Big\lfloor 25\, g\, \frac{2}{f} \Big\rfloor, \quad t_{\text{audio}} = s_0 + j \]

Next, in session 185 we put the pieces of this track together: a vision encoder, a connector and a small language model, trained end to end.

References for this session

  1. Radford et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision
  2. Xu et al. (2025). Qwen2.5-Omni Technical Report
  3. Tao et al. (2021). Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
  4. Radford et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision
  5. Xu et al. (2025). Qwen2.5-Omni Technical Report
  6. Tao et al. (2021). Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
  7. Chu et al. (2024). Qwen2-Audio Technical Report
  8. Défossez, Copet, Synnaeve and Adi (2022). High Fidelity Neural Audio Compression (EnCodec)
  9. Zeghidour et al. (2021). SoundStream: An End-to-End Neural Audio Codec
  10. Hsu et al. (2021). HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
  11. Défossez et al. (2024). Moshi: a speech-text foundation model for real-time dialogue
  12. Xie and Wu (2024). Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming
  13. Gemmeke et al. (2017). Audio Set: An ontology and human-labeled dataset for audio events
  14. Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
  15. Bai et al. (2025). Qwen2.5-VL Technical Report