Paper: MiniGPT-4 : Enhancing Vision-Language Understanding with Advanced Large Language Models
Authors: Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, Mohamed Elhoseiny
Venue: ICLR 2024
Introduction
GPT-4, released earlier, demonstrated remarkably strong multimodal capabilities.
It was able to perform tasks that previous Vision Language Models (VLMs) could not, such as looking at a hand-drawn website sketch and writing HTML code to actually implement it, or finding humorous elements in an image.
However, how GPT-4 acquired these capabilities, in terms of model architecture or training method, was not disclosed.
The authors hypothesized that GPT-4’s outstanding multimodal ability stems from a superior LLM backbone. Looking at the trajectory of LLM development, giant language models like GPT-3 were able to perform tasks that smaller language models could not.
To prove this hypothesis, the authors trained a model called MiniGPT-4, which keeps the architecture of BLIP-2, an existing open-source VLM, intact but replaces only the LLM backbone with a more advanced LLM, Vicuna. MiniGPT-4 exhibited various new capabilities that had been seen in GPT-4.
In this post, we will take a detailed look at exactly how MiniGPT-4 was trained and what performance it showed.
Main
Model Architecture
MiniGPT-4 directly adopts the vision encoder of BLIP-2.
BLIP-2 encodes an image with EVA-CLIP ViT-G/14, compresses it into 32 query embeddings via the Q-Former, and then projects them into the LLM’s embedding space through an FC layer.
MiniGPT-4 takes this trained ViT-G/14 and Q-Former as-is and freezes them, and only trains a single Linear projection layer to map their output into the LLM space.
In addition, unlike BLIP-2, which used models such as FlanT5-XXL as the LLM backbone, MiniGPT-4 uses the more advanced LLM Vicuna.
Two-Stage Training
Training proceeds in two stages: pretraining and finetuning.
First, for the pretraining stage, Conceptual Caption, SBU, and LAION were used as datasets, leveraging about 5 million image-text pairs for training.
The only thing trained in this stage is the single newly added linear projection layer. The adopted Q-Former had been tuned for FlanT5-XXL in BLIP-2, but MiniGPT-4 does not re-tune it. (Although not mentioned in the paper, the Github code confirms that a Q-Former checkpoint tuned for FlanT5-XXL was used.)
Instead, the entire job of moving its output into Vicuna’s embedding space is delegated to the projection layer, and this is used as a soft prompt for the LLM to generate the ground-truth caption.
Once pretraining is complete, additional finetuning is needed to refine the language the model generates so that it becomes more natural. The authors generated the data to be used for finetuning themselves.
For data generation, they used the model itself after it had finished pretraining. After randomly sampling 5,000 data points from the Conceptual Caption Dataset, they injected a prompt so that the pretrained model would generate detailed descriptions of those images. If a description was too short, at 80 tokens or fewer, they queried once more to continue generating.
However, the model that had only finished pretraining often included inaccurate descriptions, such as repeating a single word, producing incomplete sentences, or generating irrelevant content. Therefore, they had ChatGPT correct unnecessary repetitions and meaningless sentences. Errors that ChatGPT failed to filter out were finally verified by humans directly.
In the end, about 3,500 pairs out of the 5,000 were selected as the dataset to be used for finetuning.
Using the dataset generated this way, they carried out finetuning. The prompt injected during finetuning was as follows.
“###Human: <Img><ImageFeature></Img><Instruction>###Assistant:”
For the Instruction, they used a predefined instruction set such as “Describe this image in detail” and “Could you describe the contents of this image for me”.
When computing loss, the regression loss for this prompt was not computed; loss was computed only on the subsequent response.
This finetuning process required only 400 training steps and took just 7 minutes on a single A100 GPU.
Experiments and Evaluation
To verify MiniGPT-4’s emergent capabilities, they evaluated the ability to perform four main tasks. The four are as follows.
-
Writing advertisement copy for objects in images
-
Identifying the funny point in meme-related images
-
Generating recipes from food photos
-
Generating poems from images
For each task, evaluation was conducted with 25 images, for a total of 100 images. The evaluation was a binary classification judged directly by humans on whether the task was performed. As a result, BLIP-2 almost never succeeded, whereas MiniGPT-4 showed a 65% success rate. The figures below show the specific experimental results and example photos.
Afterward, they also conducted evaluation on standard benchmarks. They used the COCO benchmark dataset and had ChatGPT judge, with yes or no, whether the caption produced by the model contained all the information of the ground-truth caption. The prompt injected during evaluation and the results are shown in the figure below.
Ablation
Beyond measuring benchmark performance, they conducted various ablations.
First, to check whether the second finetuning stage of the two-stage training is truly effective, they compared the performance of a MiniGPT-4 model that underwent only the 1-stage pretraining with a MiniGPT-4 model that underwent both stages. As measured, the model that only underwent pretraining exhibited the various problems mentioned earlier, such as repeating a single word and incomplete sentences, and showed a clear difference in the ability to generate detailed captions and poems matching the image.
In addition, they also experimented with whether finetuning the existing BLIP-2 model, whose LLM backbone was not replaced, on the finetuning data the authors generated themselves, or training it on the Localized Narratives dataset instead of this dataset, would reach performance similar to MiniGPT-4 (MiniGPT-4 LocNa).
As a result, BLIP-2 was unable to perform tasks such as meme explanation or website coding, which demonstrated the importance of the LLM model. MiniGPT-4 LocNa was able to generate long image descriptions, but it produced low-quality answers with monotonous expressions and generalized poorly, such as not being able to explain the humorous elements of memes well.
Afterward, they also conducted ablations on the model architecture. In the MiniGPT-4 architecture, they applied three architectural changes, namely an architecture excluding the Q-Former, replacing the single linear projection layer with three layers, and training the Q-Former together in addition to the single linear projection layer, and measured the benchmark performance for each. The experimental results are shown in the table below.
These experimental results show that the Q-Former is not decisive for improving the model’s ability, and that in a limited training-data environment, a single linear projection layer is sufficient to align the vision encoder and the LLM backbone.
Limitations
MiniGPT-4 also showed clear limitations.
First, its traditional benchmark performance fell short of BLIP-2.
MiniGPT-4 (Finetune Vicuna) is a model that partially unfreezes the LLM (using LoRA) and additionally finetunes it with an extra dataset.
MiniGPT-4 lagged behind BLIP-2 on both benchmarks, and the MiniGPT-4 (Finetune Vicuna) variant showed similar performance on GQA but clearly lagged behind on the AOK-VQA benchmark.
Second is the hallucination phenomenon. As a result of experimenting with $CHAIR_i$, a metric for evaluating hallucination, hallucination appeared more than in BLIP-2.
Finally, its spatial understanding ability was also limited. For example, it could not identify whether a window is on the right or the left side of an image. The authors interpreted this as likely being because the training data lacked text-image pair datasets for spatial understanding ability.
Conclusion
MiniGPT-4 is a paper that proved that the origin of GPT-4’s emergent abilities is the use of an advanced LLM backbone. To verify this, the authors adopted BLIP-2’s vision encoder architecture and made changes to the LLM backbone, reproducing those abilities with a minimal dataset.
The authors explain the reason MiniGPT-4 came to possess these abilities as compositional generalization. The abilities GPT-4 demonstrated can in fact be seen as a combination of two fundamental abilities: image understanding and language generation.
Taking the task of writing a poem from an image as an example, an advanced LLM like Vicuna already has the ability to write a poem following the user’s instruction. Therefore, if only the ability to understand images is added on top, then even if there is no image-poem pair at all in the training data, it can generalize to the new task of “writing an image-based poem”.
The two-stage training design can also be reinterpreted from this perspective.
In the first pretraining stage, the model acquires image understanding ability by modeling the correlation between images and short captions. However, the language style of the caption dataset differs from the language the LLM originally generated, so in this process the language generation ability is distorted and the compositional generalization ability is not well expressed. The second finetuning stage plays exactly the role of restoring this language generation ability.
The fact that MiniGPT-4, having acquired both fundamental abilities, successfully generalizes to various compositional tasks such as website coding and meme interpretation supports the authors’ hypothesis.