Paper: BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Authors: Junnan Li, Dongxu Li, Caiming Xiong, Steven Hoi
Venue: ICML 2022
URL: https://arxiv.org/abs/2201.12086
Paper: BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
Authors: Junnan Li, Dongxu Li, Silvio Savarese, Steven Hoi
Venue: ICML 2023
URL: https://arxiv.org/abs/2301.12597
Paper: InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
Authors: Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, Steven Hoi
Venue: NeurIPS 2023
BLIP
Introduction
Vision-Language Pre-training (VLP) is a method that jointly learns visual and linguistic information from image-text pairs. The goal of VLP is to pre-train a model on large-scale image-text pairs and then improve performance on various downstream tasks. Such pre-trained models are used for many downstream tasks that involve both images and text, such as image retrieval, image captioning, and Visual Question Answering (VQA). However, prior work has two major limitations.
- 1. Limitations of model architecture: Existing VLMs with an Encoder or Encoder-Decoder architecture were biased toward only one of the two areas: the understanding area, which predicts text from an image, or the generation area.
- 2. Limitations of training data: Because high-quality datasets written by humans are expensive to collect, most work relied on web data, but web data was frequently noisy.
BLIP addresses the above limitations by introducing MED (Multimodal Mixture of Encoder-Decoder) and CapFilt.
MED (Multimodal Mixture of Encoder-Decoder)
MED consists of three structures. For each batch, it uses the image features extracted by the Image Encoder to compute a loss for each text encoder or decoder, and optimizes them simultaneously.
1. Image-Text Contrastive Loss (ITC) Using a unimodal text encoder, the ITC loss is computed based on the similarity between the image information and the text information obtained from the text encoder. It uses contrastive learning that maximizes the similarity of matched (positive) pairs and minimizes the similarity of the remaining (negative) pairs.
2. Image-Text Matching Loss (ITM) Using an image-grounded text encoder, the model learns via binary classification whether a given pair is a matched pair. It applies hard negative mining to finely learn the difference between positive pairs and hard negative pairs.
3. Language Modeling Loss (LM) Using an image-grounded text decoder, the model is trained to generate the correct text for an image. Here, the decoder is trained to maximize the log likelihood of the ground-truth caption.
CapFilt
CapFilt uses its two core structures, the Captioner and the Filter, to filter out noisy data pairs in which the image and text information do not match.
The function of each structure is as follows:
- Captioner: Generates a synthetic caption $T_S$ for an input web image $I_W$.
- Filter: Takes as input both the synthetic dataset $(I_W, T_S)$ and the web dataset $(I_W, T_W)$, and for each data pair in the datasets uses the ITM Head to judge whether the image and text match.
An example of an image produced after passing sequentially through the Captioner and the Filter is shown below.
For a single image, the red text is noisy text and the green text is correctly matched text. When an image is paired with noisy text, that pair is removed by the Filter.
After removing noisy data with the Filter, MED is pre-trained from scratch on the final dataset $D$, formed by combining the result with a high-quality human-labeled dataset.
Experimental Results
Experimental Setup
- Model architecture: For MED’s Image Encoder, ViT-B/16 or ViT-L/16 pre-trained on ImageNet is used, and BERT-base is used as the Text Encoder.
- Pre-training dataset: Two settings, 14M/129M, are used: a 14M setting consisting of COCO, Visual Genome, CC3M, CC12M, and SBU, or a 129M setting that additionally uses 115M taken from LAION 400M. Both settings are used in the experiments.
MED Parameter Sharing BLIP’s MED architecture shares all parameters except those of the Self-Attention layers. This is because, for layers other than the Self-Attention layers, the roles performed in the Encoder and the Decoder are similar.
Conversely, the role of the Self-Attention layer differs between the text encoder and the text decoder, which can cause conflicts during training. Therefore, the parameters of the Self-Attention layers are not shared.
In addition, both the model’s understanding performance (Retrieval) and text generation performance (Caption) were highest when only the Self-Attention layers were not shared.
Effect of Caption Diversity The Captioner generates captions using nucleus sampling.
This approach is more likely to generate noisy text than beam search, but it is used because the benefit gained from generating more diverse captions is greater.
The paper also examines the parameter-sharing scenario within the CapFilt structure. As shown in the table below, when the Captioner and Filter share parameters, the Filter is less likely to screen out noisy text among the synthetic text generated by the Captioner; this shows that each structure should be fine-tuned independently.
BLIP-2
Introduction
Rather than building a single large multimodal model, a number of studies have sought to leverage untrained frozen models in order to cost-effectively retain the strong performance of each unimodal model.
However, because most of them were used mainly for text generation tasks, it was difficult to sufficiently bridge the modality gap between the vision model and the LLM.
BLIP-2 resolves the problems arising from the use of frozen models by placing a Querying Transformer (Q-Former) that uses learnable query parameters between the frozen image encoder and the frozen LLM.
The pre-training process of the Q-Former consists of two stages.
- Stage 1: Learns the part of the image information obtained from the frozen Image encoder that is semantically connected to text, using a fixed-length Query parameter.
- Stage 2: Trains the Q-Former by feeding the Query output obtained in Stage 1 into the frozen LLM.
Q-Former Architecture
Stage 1: Vision-Language Representation Learning
This is the stage in which the Q-Former is trained to extract the image information most similar to the text. Each structure comprising the Q-Former shares the Self-Attention layers and performs the following functions:
- Image Transformer: Using the learnable Query parameters inside the Q-Former and the image embeddings obtained from the frozen Image encoder, it outputs a Query output that references image information.
- Text Transformer: Can be used as either a text encoder or a text decoder depending on the training objective. Here, as shown below, it uses a different Attention Mask for each training objective to control the flow of information between the Query and the text.
Each loss function is composed as follows:
1. Image-Text Contrastive Learning (ITC) The loss is computed by the same mechanism as BLIP’s ITC loss. However, the value used as the similarity differs: it uses the highest value among the image-text similarities computed for each Query. In this case, a Unimodal Attention Mask is used so that the Query and the text cannot reference each other during training.
2. Image-Text Matching (ITM) Based on the similarity matrix obtained during the ITC loss computation, the loss is computed by the same mechanism as BLIP’s ITM loss. A Bidirectional Attention Mask is used so that the Query and the text can reference each other’s information during Self-Attention.
3. Image-grounded Text Generation (ITG) The Causal Attention Mask ensures that, during Self-Attention, the Query cannot reference the text information, and that the text can reference only previously generated tokens. The model is trained to extract enough information from the input image to generate the text from scratch.
Stage 2: Vision-to-Language Generative Learning
The Query obtained in Stage 1 is used as text containing image information (a soft visual prompt) and is prepended to the LLM’s input text so that they are fed in together. The LLM can then generate the next token by referring to the image information contained in the Query.
The Stage 2 architecture is divided into a Decoder-based LLM (OPT) and an Encoder-decoder based LLM (FlanT5) depending on the LLM used, and both are trained with a Language Modeling loss.
Experimental Results
Experimental Setup
-
Model architecture: The frozen Image Encoder uses either the CLIP ViT-L/14 or the EVA-CLIP ViT-g/14 model, and the Q-Former is a Transformer with 188M parameters initialized with the weights of BERT-base.
-
Pre-training data: BLIP-2 uses the 129M-scale setting from among BLIP’s settings.
Zero-Shot image-to-text generation By changing the instruction, the model’s zero-shot performance is evaluated on tasks such as image captioning, image-text retrieval, VQA, and instructed image-to-text generation.
These experimental results confirm that BLIP-2 achieves higher zero-shot performance than other models while training the fewest parameters compared to existing models.
InstructBLIP
Introduction
Existing zero-shot image-to-text generation methods such as BLIP-2 do not consider the instruction when extracting image information. As a result, they extract the same information from an image regardless of the user’s request and pass it to the LLM.
InstructBLIP trains a pre-trained BLIP-2 with the user’s question or prompt itself, so that the information extracted from the image varies according to the user’s request.
Vision-Language Instruction Tuning
In BLIP2, after the Q-Former was trained, the learned Query was used for the LLM task. InstructBLIP fine-tunes the already-trained Q-Former from BLIP2 with instructions so that the Query learns instruction information.
The Query is then used as a soft visual prompt placed before the instruction in the LLM, enabling it to effectively learn the instruction information.
Dataset
InstructBLIP classifies 26 datasets into 11 tasks and converts them into an instruction tuning format for use. The datasets can be classified as follows:
- held-in datasets (13): shown in yellow in the figure, used for instruction tuning.
- held-out datasets (13): shown in white in the figure, used for evaluation.
During instruction tuning, using 10-15 human-written natural language instruction templates per task, the Query can learn the user’s request on a per-task basis.
Afterward, the text of each held-out dataset used for evaluation is converted into the following form depending on the dataset.
Training Details
To address overfitting or underfitting caused by size differences between datasets, during training a dataset $d$ is sampled with probability $p_d = \frac{\sqrt{S_d}}{\sum_{i=1}^{D} \sqrt{S_i}}$, that is, with a probability proportional to the square root of the dataset size.
In addition, at inference time, two generation methods are used depending on the dataset characteristics:
- Datasets corresponding to image captioning or open-ended VQA tasks: the model directly receives the instruction and generates a response.
- Datasets corresponding to classification or multi-choice VQA tasks: using the vocabulary ranking method, the candidate answer with the highest log-likelihood among the candidates is selected as the answer.
Zero-shot Evaluation
Comparing zero-shot performance against the existing SOTA models BLIP-2 and Flamingo, InstructBLIP achieves a new SOTA on all held-out datasets.
The paper also shows that instruction tuning is key to the high zero-shot generalization performance. As shown below, in a performance evaluation according to whether Multitask learning and Instruction Tuning were used, we can confirm that the model’s zero-shot generalization performance was highest when trained with Instruction Tuning (InstructBLIP).
Conclusion
From BLIP to InstructBLIP, VLMs have evolved beyond generating captions for images toward understanding user input and generating answers that match the request. The BLIP model jointly learned understanding and generation capabilities between text and image data, and BLIP-2 proposed an architecture that trains only lightweight modules on top of well-established SOTA models. InstructBLIP went further and trained the model to better understand the user’s request. In other words, the number of parameters the model needs to learn kept decreasing while the range of tasks it can perform expanded.
In InstructBLIP, we could confirm the perspective that, in the BLIP-2 architecture, which part of the image to extract information about depends on the user’s request. This perspective, that the model’s perception cannot be separated from the user’s intent, remains valid even as later models have improved to simpler forms and stronger performance.