Paper: Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
Authors: Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, Jingren Zhou
Venue: arXiv
URL: https://arxiv.org/abs/2308.12966
Paper: Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
Authors: Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, Junyang Lin
Venue: arXiv
URL: https://arxiv.org/abs/2409.12191
Paper: Qwen2.5-VL Technical Report
Authors: Qwen Team, Alibaba Group
Venue: arXiv
URL: https://arxiv.org/abs/2502.13923
Paper: Qwen3.5-Omni Technical Report
Authors: Qwen Team
Venue: arXiv
Qwen-VL
Introduction
Existing Large Vision-Language Models (LVLMs) were able to generate image descriptions or question answers by connecting a Vision Encoder to a pre-trained language model (LLM). At the time, however, most open-source models focused on capturing the overall meaning of an image and showed limitations in fine-grained visual perception, such as reading small text within an image or locating an object referred to by the user.
Qwen-VL aims to connect visual input to Qwen-7B and perform the following vision-language capabilities within a single model.
- General image captioning and Visual Question Answering (VQA)
- Text-oriented VQA that reads characters and documents inside images
- Visual Grounding that locates a specific object by coordinates or describes the object indicated by given coordinates
- Multi-image comprehension that understands inputs where several images are interleaved with text
Model Architecture
Qwen-VL consists of three modules: Visual Encoder, Position-aware VL Adapter, and QwenLM.
- Visual Encoder: A 1.9B-scale Vision Transformer initialized from OpenCLIP’s ViT-bigG. It divides the image into 14×14 patches to generate a visual feature sequence.
- Position-aware VL Adapter: Composed of a single 0.08B-scale Cross-Attention Layer. 256 learnable Queries attend to the full features of the Visual Encoder, compressing the long image features into 256 visual tokens.
- QwenLM: A 7.7B-scale language model initialized from Qwen-7B. It takes visual tokens and text tokens as a single sequence and generates the next token.
The VL Adapter is not a simple projection layer. A 448×448 image becomes 1,024 patch features after passing through the ViT, and feeding these directly into the LLM greatly increases the amount of computation. The Adapter uses 256 Queries to compress this sequence into a fixed 256 tokens, thereby reducing the LLM’s input length and computational cost.
Position-aware VL Adapter
In the Adapter’s Cross-Attention layer, the Query consists of 256 Learnable Query Embeddings, while the Key and Value are the visual features generated by the ViT. Since each Query attends to the patch features of the entire image, the output length is always fixed at 256.
However, during compression, the information about which position in the image each feature came from can be diluted. To prevent this, Qwen-VL adds 2D absolute positional encoding to the Query and Key.
\[\begin{equation} \tilde{Q} = Q + P_{\mathrm{query}}, \qquad \tilde{K} = K + P_{\mathrm{image}}, \label{eq:qwen_vl_position_encoding} \end{equation}\]The positional information is applied only to the Query and Key used to compute the Attention score, and is not added to the Value. As a result, the Adapter can select which visual information is important while also taking into account where in the image that information came from.
Input-Output Interface
An image is converted into 256 visual tokens through the Visual Encoder and the Adapter. The model adds special tokens <img> and </img> before and after these tokens to distinguish text tokens from image tokens, and feeds the entire sequence into QwenLM.
Qwen-VL also treats Grounding not as a separate Detection Head but as a language generation problem. It normalizes bounding box coordinates to the $[0,1000)$ range and places them as text between special tokens. In this way, the model can generate the location corresponding to a specific expression as coordinates, or describe the object indicated by given coordinates in a sentence.
Three-Stage Training
Qwen-VL goes through a total of three training stages as follows.
1. Pre-training
In the first stage, low-resolution 224×224 image-text pairs are used. QwenLM is kept frozen while the Visual Encoder and VL Adapter are trained, so that image features are first aligned to the already-learned language space. As the training dataset, about 1.4B pairs, refined from 5B raw image-text pairs, are used.
2. Multi-task Pre-training
In the second stage, the input resolution is raised to 448×448 and the entire model is trained. Seven tasks, including Captioning, VQA, Grounding, OCR, and Pure-text Autoregression, are used together to learn more fine-grained information such as small text and object locations.
3. Supervised Fine-tuning
In the final stage, the Vision Encoder is frozen while the VL Adapter and QwenLM are fine-tuned on instruction data to produce Qwen-VL-Chat. By using Grounding and multi-image dialogue data together with caption and dialogue data, the model can go beyond simply describing an image and respond in the format requested by the user.
Experimental Results
Qwen-VL shows higher performance than similarly sized Generalist Models not only on Image Captioning and VQA but also on text-centric tasks such as TextVQA, DocVQA, ChartQA, and OCR-VQA, and on RefCOCO-family Grounding tasks. In particular, it substantially outperformed the open Generalist Models of the time on Flickr30K zero-shot captioning and on VQAv2, OKVQA, and GQA, and Qwen-VL-Chat’s actual conversational instruction-following ability is confirmed through TouchStone, SEED-Bench, and MME.
Number of Learnable Queries
If the Adapter has too few Queries, visual information is lost during compression, whereas too many Queries increase the LLM’s input length and training difficulty. In this paper, 64, 144, 256, and 400 Queries are compared, and 256 is chosen by considering information preservation, convergence speed, and computational cost. This experiment indicates that the number of visual tokens is not merely a hyperparameter but a bottleneck that determines the compression rate of visual information.
Qwen2-VL
Introduction
Qwen-VL focused on high-resolution images and fine-grained visual information, but the constraint of converting every image to a fixed 448×448 size remained. Fitting images with different original aspect ratios and resolutions into a single size unnecessarily enlarges small images, while large images or long documents can lose detail.
Limitations also exist in video processing. A video is a continuous sequence of images that change over time, but existing models either treated video as a separate modality or used only 1D positional information. With such approaches, however, it is difficult to simultaneously represent the temporal change in a video and the 2D positions within a frame.
Qwen2-VL maintains the existing Vision Encoder–Connector–LLM structure while focusing on the following three changes.
- Naive Dynamic Resolution, which produces a different number of visual tokens for each image
- Multimodal Rotary Position Embedding (MRoPE), which unifies the positions of text, images, and video
- Unified Image and Video Understanding, which processes images and video with the same Vision Transformer
Model Architecture
Qwen2-VL uses a Vision Transformer of about 675M scale in common across all sizes, and is divided into 2B, 7B, and 72B models depending on the size of the Qwen2-family LLM. An image is fed into the ViT while preserving its native resolution and aspect ratio, and the visual features pass through a 2×2 Patch Merger. The Patch Merger merges four adjacent tokens into one while projecting them into the LLM’s hidden dimension.
For example, dividing a 224×224 image with a patch size of 14 produces 16×16=256 patch tokens. After 2×2 merging, 64 visual tokens remain, and including the surrounding <vision_start> and <vision_end> (special tokens), a total of 66 tokens are fed into the LLM.
Naive Dynamic Resolution
Dynamic Resolution does not restrict all images to the same size. The number of patches varies with the original size and aspect ratio of the image, and as a result the number of visual tokens fed into the LLM varies as well. A large image with much detail uses more tokens, while a simple or small image can be processed with fewer tokens.
However, the fixed absolute position embedding of the existing ViT presumes a patch grid of a predetermined size, so it has the limitation of being unable to directly process inputs whose grid size changes with each resolution. To resolve this, the paper removes the fixed absolute position embedding and applies 2D-RoPE. Since 2D-RoPE reflects the height and width coordinates of each patch into the Attention Query and Key, it can represent the relative spatial structure even when the size of the input grid changes.
Multimodal Rotary Position Embedding (MRoPE)
The 1D-RoPE of existing LLMs represents only the order of tokens. Qwen2-VL divides the feature dimension of the Query and Key into three parts, Temporal, Height, and Width, and applies a different position ID to each part.
- Text: Uses the same position ID across the three axes, so it behaves identically to ordinary 1D-RoPE.
- Image: Keeps the Temporal ID the same while changing the Height and Width IDs according to patch position.
- Video: The Temporal ID increases with each frame, and within each frame the Height and Width IDs vary with patch position.
MRoPE not only explicitly separates the spatial and temporal information of images and video, but also reduces the growth rate of the position IDs. In the paper, even though the video length was limited to 16K tokens during training, the model shows stable performance in the range of up to 80K tokens at inference, confirming the potential for length extrapolation on long videos.
Unified Image and Video Understanding
Qwen2-VL’s Vision Transformer uses 3D Convolution to process images and video in the same input format. Video is patchified into tube units with a temporal depth of 2, and a still image is duplicated into two frames to apply the same operation.
Video is sampled at 2 frames per second, and to control the computation for long videos, the number of visual tokens per video is limited to at most 16,384. Through this design, images and video are processed within a single Vision Transformer and positional representation system, rather than being split into separate encoders.
Three-Stage Training
Qwen2-VL also maintains the three-stage training scheme of Qwen-VL.
1. Vision Pre-training
In the first stage, the Qwen2 LLM is frozen and training centers on the ViT to learn basic image-text relationships, OCR, and image classification ability.
2. Multimodal Pre-training
In the second stage, the entire model is trained using VQA, interleaved image-text, video dialogue, and pure text data together. The tokens used in the two pre-training stages total 1.4T in scale, and the loss is computed only at text token positions, not at visual tokens.
3. Instruction Fine-tuning
In the final stage, the ViT is frozen and training centers on the LLM to learn Image QA, document parsing, multi-image comparison, video understanding, and Agent Interaction.
Qwen2-VL unifies all tasks into a text-generation format. Grounding generates normalized coordinates, and the Visual Agent observes the screen and then selects a permitted Action as text.
Experimental Results
Qwen2-VL is evaluated broadly across General VQA, document and chart understanding, Multilingual OCR, mathematical reasoning, video understanding, and Visual Agent. The 72B model records high performance among the open Generalist Models of the time on many benchmarks such as DocVQA, InfoVQA, TextVQA, OCRBench, and MathVista, and shows results comparable to GPT-4o and Claude 3.5 Sonnet on several items.
Dynamic Resolution Ablation
Comparing a setting that uses a fixed number of image tokens against Dynamic Resolution, no single fixed resolution was best across all benchmarks. Dynamic Resolution maintained top-tier performance while using fewer tokens on average. Higher resolution was not always better either, and excessive enlargement could push small images out of the training distribution and lower OCR performance.
MRoPE Ablation
Comparing 1D-RoPE and MRoPE, MRoPE maintained overall competitive performance on several image tasks while showing more pronounced improvements on video benchmarks such as PerceptionTest, NextQA, and STAR.
Model Scaling
Comparing the 2B, 7B, and 72B models along with the scale of training data, the larger the model and data, the more consistently performance improves across document understanding, mathematical reasoning, general VQA, and video understanding. In this paper, it is experimentally confirmed that, as with LLMs, scaling of model and data is also effective in LVLMs.
Qwen2.5-VL
Introduction
Qwen2-VL could process images of various resolutions, but the coordinates the model output did not actually reflect the real size of the image. Video, too, was sampled at a fixed frame rate, and since the temporal position ID increased with the frame index, absolute time could not be taken into account.
Qwen2.5-VL represents spatial information as real pixel coordinates and temporal information as position IDs aligned to the real elapsed time, so that it perceives spatial scale and temporal flow directly, without any separate normalization.
In this section, we examine how Qwen2.5-VL converted space and time into absolute representation, how the vision encoder was redesigned to handle the accompanying computation, and what results the training at a scale of 4.1 trillion tokens led to.
Absolute Representation
Spatial Representation
When outputting the coordinates of a grounding box, Qwen2-VL normalized them to the $[0, 1000)$ range. This approach represents all images in the same coordinate system, so the object’s actual size and position do not remain in the coordinates.
Qwen2.5-VL directly outputs the pixel coordinates corresponding to the actual width and height of the input image. The training data is likewise constructed on the same principle: by constructing the grounding coordinates with respect to the actual image size, the model is made to learn the actual scale and spatial relationships of objects directly. As a result, it supports not only bounding boxes but also point-form grounding that indicates detailed locations hard to express as a box, and it can stably generate outputs in structured formats such as XML and JSON.
Temporal Representation
MRoPE divides the position embedding into three components: time, height, and width. Text uses the same ID for all three components, behaving identically to 1D RoPE; images keep the time ID fixed while only the spatial IDs vary; and video increases the time ID for each frame.
In Qwen2-VL, the time ID was tied to the number of input frames and thus could not represent absolute time intervals.
Qwen2.5-VL assigns the position ID of the time component according to the actual timestamp rather than the frame order. Frames at the same instant receive the same ID no matter what FPS they are sampled at, and because the ID intervals reflect real time, the model can learn absolute time.
Dynamic FPS Sampling dynamically varies the FPS per video during training so that the time intervals the model perceives do not depend on a particular FPS.
This approach carries information in the intervals of the already-existing position IDs, conveying temporal information through the position ID intervals alone, without additional computation.
Computation Optimization of the Vision Encoder
Window Attention
Among the ViT layers, only layers (7, 15, 23, 31) perform full attention, while the remaining 28 layers perform attention only within a window of at most 112x112 pixels. Regions smaller than the window are processed as-is without padding, so the original resolution is not distorted. In the window attention configuration, the computation grows linearly rather than quadratically with the number of patches, and the 4 full-attention layers learn the overall context to prevent information loss.
The ViT structure was also reconstructed, improved to a structure using RMSNorm, SwiGLU, and 2D-RoPE, and trained from scratch on DataComp and in-house data instead of using existing weights.
Vision-Language Merger
The ViT output goes through a Merger that concatenates the patches of adjacent 2x2 regions and passes them through a 2-layer MLP before entering the LLM. The visual features that pass through the Merger are projected into the same space as the text embeddings, and by concatenating patches for computation, the number of patches is reduced to 1/4. In other words, computation is reduced by window attention in the vision encoder and by the Merger in the LLM.
Training
The scale of pre-training data increased from Qwen2-VL’s 1.2 trillion tokens to 4.1 trillion tokens. The types were also expanded, using document omni-parsing that unifies and parses tables, charts, formulas, chemical formulas, and musical scores into HTML format, video grounding that outputs videos on a per-second basis, and agent data that operates computers and mobile devices.
Pre-training proceeds in three stages:
- Visual Pre-Training: With the LLM frozen, only the ViT is trained on 1.5 trillion tokens to align it with the language model.
- Multimodal Pre-Training: The full parameters are trained on 2 trillion tokens of interleaved data, VQA, video, agent, and so on.
- Long-Context Pre-Training: Long documents and long-duration videos, with sequence length increased by about 4×, are trained.
Post-training proceeds in two stages, SFT and DPO, and the ViT is frozen in both stages. For tasks requiring multi-step reasoning such as math and code, rejection sampling was used to keep only CoT that matches the correct answer.
Experimental Results
The flagship 72B model is on par with GPT-4o, Claude 3.5 Sonnet, and others on general benchmarks such as MMMU and Math Vista.
On the other hand, it surpasses existing SOTA models on documents and OCR. On CC-OCR it substantially outperforms GPT-4o and Gemini 1.5 Pro, and it also shows excellent performance on OCRBench_v2.
On video grounding, it substantially outperforms GPT-4o on Charades-STA mIoU. However, this figure reflects both data expansion and the LLM backbone, so it is not a result achieved by the paper’s proposed absolute-time alignment alone. The change in Agent is the most striking, dramatically raising the performance of Qwen2-VL-72B on ScreenSpot Pro.
Qwen3.5-Omni
Introduction
Qwen3.5-Omni has been extended to understand the four modalities of text, image, video, and audio within a single model, and to respond not only in text but also in speech. On top of Qwen3-Omni’s Thinker and Talker structure, it uses Qwen3.5’s Hybrid-Attention MoE Transformer, and by introducing a 256K context and ARIA, it enables the processing of long-duration audio-video input and real-time speech generation. Below, we examine how Qwen3.5-Omni unifies and understands different modalities and generates real-time speech.
Interleaved input sequence
Each modality is processed by an individual encoder and then combined into a single sequence.
- Text: Qwen3.5 tokenizer
- Image, Video: Uses Qwen3.5’s SigLIP2 Vision Encoder, sampled at dynamic frames to align with audio.
- Audio: Resampled to 16kHz, converted to a mel-spectrogram, and encoded with an Audio Transformer (AuT). A single output token corresponds to about 160ms of the original signal.
For video input, visual tokens and audio tokens are interleaved in temporal order, and a per-second timestamp is inserted as a string before each temporal patch to represent the temporal relationship between video and audio.
Here the position ID is kept at a constant resolution of 160ms units, and absolute time is specified by the text timestamp. This is because using the position ID directly as absolute time would make the ID grow excessively large and sparse for long video inputs, weakening long-range temporal modeling, and would require large-scale uniformly sampled data across various FPS, raising the construction cost.
Real-Time Speech Streaming
- Thinker: Takes the unified sequence, understands and reasons over it, and then generates a text response autoregressively.
- Talker: Takes the Thinker’s output, the multimodal context, and the user’s input, and generates RVQ codec tokens autoregressively in real time.
Text tokens and codec tokens are not generated at the same rate. For example, a single text token may be a short syllable or a word pronounced with several syllables, so mechanically interleaving text and speech one-to-one can cause problems such as dropped words and pronunciation errors.
ARIA (Adaptive Rate Interleave Alignment) dynamically adjusts the correspondence rate between text units and speech units during the Talker’s codec token generation, and then unifies them into a single stream.
Specifically, during training it computes the global ratio, which is the ratio between text and codec tokens, so that when generating codec tokens the ratio of the accumulated number of text tokens $N_{text}$ to the number of codec tokens $N_{speech}$ does not exceed the global ratio. \(\begin{equation} N_{speech}\over{N_{text}} \le \texttt{global ratio} \end{equation}\) As a result, it can generate more stable and natural streaming speech while preserving the textual meaning of the Thinker.
The RVQ codec tokens generated in real time by the Talker are used in MTP to generate the remaining codec tokens based on the codebook, and in Code2Wav to synthesize the waveform on a per-frame basis and output the final speech.
Experimental Results
Now capable of real-time speech processing, Qwen3.5-Omni measures speech generation performance under various settings. Here, depending on the difference between the Thinker and Talker, performance is measured on two models, Qwen3.5-Omni-Flash and Qwen3.5-Omni-Plus.
In this article, we focus mainly on the model’s speech generation performance.
- zero-shot voice cloning: Checks whether, given a voice and text it has never learned, the model reads the text accurately and without errors in the given voice.
- multilingual speech generation: Measures the extensibility of zero-shot voice cloning across 29 languages.
- cross-lingual speech generation: Evaluates whether, given a voice it has never learned and text in a different language, the model reads the text accurately and without errors in the given voice.
- custom-voice speech generation: After fine-tuning on one of the voices learned in the Talker’s system prompt, evaluates whether it reads text in 29 languages accurately and without errors in that voice.
Each task is measured with the following two metrics.
- WER (Word Error Rate): Converts the speech generated by the model into text with an ASR model to measure whether any words are dropped or distorted. GPT-4o-transcribe was used as the ASR model.
- SIM: Measures the similarity between the speech generated by the model and the target speech using cosine similarity.
Having shown high zero-shot speech generation performance on the SEED-TTS benchmark, Qwen3.5-Omni, as shown in the table above, also shows relatively high performance across 29 languages in comparison with various commercial models. Moreover, even when pronouncing a language different from that of the given voice, it stably preserves the speaker’s voice and shows outstanding speech generation ability compared with existing benchmarks and commercial models.
Conclusion
The evolution of the Qwen multimodal series follows the question of “in what coordinate system should the model view the world.” Whereas Qwen-VL and Qwen2-VL addressed the problem of escaping the constraint of fixed resolution to handle arbitrary-resolution inputs, Qwen2.5-VL sought to represent space and time absolutely, and Qwen3.5-Omni added speech on top of this to unify four modalities into a single sequence, broadening the scope beyond understanding to real-time speech generation and interaction.