📖Flamingo Series: Flamingo, OpenFlamingo

Paper: Flamingo: a Visual Language Model for Few-Shot Learning

Authors: Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikołaj Bińkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, Karen Simonyan

Venue: NeurIPS 2022

URL: https://arxiv.org/abs/2204.14198

---

Paper: OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models

Authors: Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, Ludwig Schmidt

Venue: arXiv preprint (arXiv:2308.01390), 2023

URL: https://arxiv.org/abs/2308.01390

---

Flamingo

Introduction

The ability to quickly perform a new task from only a short instruction is an important characteristic of intelligence. However, existing computer vision models often demand large amounts of training data and separate fine-tuning for each new task, which is costly. Existing vision-language models that align images and text through contrastive learning likewise face the limitation that they are difficult to apply to open-ended tasks that generate free-form text.

To address these limitations, Flamingo, which can quickly adapt to a wide range of vision-language tasks from only a handful of multimodal examples, was proposed. Flamingo takes as input a sequence in which images or videos and text are interleaved, and generates text autoregressively conditioned on the visual information. In particular, without fine-tuning the model’s parameters for each new task, it can perform the task through few-shot in-context learning alone, by presenting a few examples in the prompt.

The problem Flamingo must solve here is not simply connecting a vision encoder and a language model. While injecting visual information into an already sufficiently pretrained language model, it must not damage the model’s existing language ability. Flamingo’s frozen backbone, Perceiver Resampler, and GATED XATTN-DENSE are the components that solve this problem.

Flamingo Architecture

Flamingo is fundamentally an autoregressive model that predicts the next text token. Like a typical language model, it uses previous text tokens, but it additionally conditions on the images and videos presented up to the current position.

Looking first at the overall structure, the NFNet-F6 vision encoder, pretrained with contrastive learning, and the Chinchilla-family pretrained language model are kept frozen. Only the Perceiver Resampler and GATED XATTN-DENSE that connect the two models are newly trained.

Looking at the details, when image and video inputs first arrive, the Frozen Vision Encoder converts those pixels into visual features. Images passed through the Frozen Vision Encoder are converted into a 2D spatial feature grid. Videos are encoded frame by frame independently, after which a learnable temporal embedding is added to represent spatiotemporal information. The spatial dimensions are then flattened to construct a variable-length visual feature sequence.

For images, as many visual features are produced as the number of spatial grid cells; for videos, as many as the product of the number of frames and spatial grid cells. Feeding these features directly into the language model’s cross-attention would greatly increase the number of visual tokens and the amount of computation. To alleviate this problem, the Perceiver Resampler converts the variable-length visual features into a fixed set of 64 visual tokens per image or video. To do so, it uses 64 learnable latent queries. The Query is the latent $Z$, and the Key and Value are $[X_f;Z]$, formed by concatenating the visual features $X_f$ and the latents.

Here, the 64 tokens are not 64 specific patches selected from the image, but the result of summarizing the entire visual input into 64 learned representations. By fixing the number of visual tokens, the computational cost of the subsequent vision-language cross-attention can be bounded.

The visual tokens output in this way are reflected into the text at the GATED XATTN-DENSE layer. This layer is inserted between the frozen language model blocks. In the cross-attention, the Query is the text hidden state $Y$ produced by the language model, and the Key and Value are the 64 visual tokens $X$. Here, GATED refers to a learnable gate parameter:

Looking at the equations, we can identify $\alpha_{\mathrm{xattn}}$ and $\alpha_{\mathrm{dense}}$. These gate parameters are initialized to 0 at the start of training. Because the newly added cross-attention and feed-forward layers are initially randomly initialized, adding their output strongly to the frozen LM’s hidden state from the very beginning could disturb the existing representations and destabilize training. Early in training, the added layers’ output does not affect the residual stream, and as training progresses, the degree to which visual information is reflected is gradually adjusted.

The insertion frequency of GATED XATTN-DENSE varies with model size. Flamingo-3B inserts it before every LM block, but for larger models, considering computational cost, Flamingo-9B inserts it every 4 blocks and Flamingo-80B every 7 blocks. The detailed settings are as follows.

For inputs containing multiple images, Per-image Attention Masking limits which visual tokens each text token can directly attend to.

Each text token directly cross-attends only to the most recent image or video among the visual inputs that appeared before it. For example, the text span after the first image directly attends to the first image, and the text span after the second image directly attends to the second image.

That does not mean the information from previous images disappears entirely. Because the information from the first image is already reflected in the hidden state of the first text span, the subsequent text can indirectly receive that information through the language model’s causal self-attention.

This scheme was selected through an ablation study. Directly attending only to the single most recent image was 7.2% better in overall score than directly cross-attending to all previous images. Furthermore, although training used at most 5 images per sequence, at evaluation time performance improved even when the number of image/video-text pairs was increased up to 32.

Training Dataset

Flamingo is trained by jointly using four vision-language datasets with different input formats. The training data is broadly divided into interleaved image-text data collected from web pages, general image-text pairs, and video-text pairs.

In particular, M3W (MultiModal MassiveWeb), in which multiple images and text are naturally interleaved, plays an important role in enabling Flamingo to learn the multimodal in-context learning format. Pair-form data is also converted to have the same input format as M3W, as follows.

<image> caption <EOC>

Through this, the four datasets with different formats can all be trained under a single autoregressive language-modeling objective.

Training Procedure

For each dataset, Flamingo predicts the next text token conditioned on the previous text tokens and the visual inputs given up to the current position. At each token position, it computes the negative log-likelihood of the probability assigned to the correct token, and sums this over the entire sequence. Because each dataset differs in size and role, a per-dataset weight $\lambda_m$ is also applied. Flamingo’s overall training objective is as follows.

Here, $\mathcal{D}m$ is the $m$-th dataset, $y\ell$ is the correct text token at the current position, $y_{<\ell}$ is the previous text tokens, and $x_{\leq\ell}$ is the visual input given up to the current position. If the model assigns a high probability to the correct token, the loss becomes small; if it assigns a low probability, the loss becomes large.

During training, the Vision Encoder and Language Model are kept frozen. Therefore, the parts that are actually trained are the Perceiver Resampler, which converts variable-length visual features into 64 visual tokens, and the GATED XATTN-DENSE layers, which inject visual information into the language model.

Evaluation Method

To verify whether a single model can adapt well to a variety of image and video tasks, Flamingo was evaluated on a total of 16 benchmarks including captioning, visual question answering, visual dialogue, multiple-choice question answering, and multimodal classification.

Among these, 5 benchmarks—COCO, OKVQA, VQAv2, MSVDQA, and VATEX—were used as DEV benchmarks for model design and hyperparameter decisions. The remaining 11 benchmarks were not used for model design decisions and served to evaluate general performance.

Few-shot evaluation proceeds by first presenting several support examples and then placing at the end a query for which the answer is not given.

[Image/Video + Task input + Answer] × N
                         ↓
          [Query image/video + Task input]
                         ↓
                    Prediction

In this process, the model’s parameters are not updated to fit the downstream task. The paper reports its main results under 0-shot, 4-shot, and 32-shot conditions. The evaluation method is divided into two types depending on the output form of the task.

• Open-ended evaluation: the model directly generates free-form text. By default it uses beam search with beam size 3, and terminates the output when the <EOC> token is generated.
• Close-ended evaluation: each possible candidate answer is appended after the prompt to compute its log-likelihood, and the answer with the highest score is selected.

Main Results

Flamingo surpassed previous zero-shot and few-shot methods by a large margin on 16 benchmarks. In particular, on all 9 benchmarks for which prior few-shot results were available, it newly achieved the few-shot state of the art of the time.

As the model size and the number of demonstrations increased, overall performance also improved. Flamingo-9B outperformed Flamingo-3B, and the largest Flamingo-80B outperformed Flamingo-9B in few-shot performance; the larger the model, the more effectively it leveraged many demonstrations such as 32-shot.

Representatively, the COCO captioning and VQAv2 results show the change according to the number of shots.

An interesting result is that Flamingo-80B, using only 32 task-specific examples and without any separate downstream parameter update, surpassed on 6 tasks even the performance of prior methods fine-tuned on thousands or more annotated samples.

Although the paper’s main interest is in-context learning without parameter updates, the authors additionally conducted experiments directly fine-tuning Flamingo when sufficient annotated data was available. As a result of fine-tuning Flamingo-80B on the 9 tasks where few-shot alone had not surpassed the prior fine-tuned SOTA, it achieved a new SOTA on 5 tasks—VQAv2, VATEX, VizWiz, MSRVTTQA, and HatefulMemes.

The ablation study reports the results of evaluating Flamingo-3B in the 4-shot setting on the 5 DEV benchmarks. Summarizing only the main results:

The largest change occurred when the interleaved image-text dataset M3W was removed. Removing the general image-text pairs also decreased the score, and removing the video-text pairs lowered performance on video tasks overall. This showed that naturally interleaved data and large-scale pair data serve different roles.

In addition, the zero-initialized tanh gating, per-dataset gradient accumulation, Perceiver Resampler, and frozen LM each actually contributed to performance. Ultimately, Flamingo’s performance was greatly influenced not only by model size, but also by which training data was used and how the visual information was connected to the existing language model.

Conclusion

Flamingo adds a Perceiver Resampler and GATED XATTN-DENSE between a pretrained Vision Encoder and Language Model to generate text based on images and videos, without greatly altering the model’s existing language ability.

The most important point is that the model does not need to be fine-tuned again for each new task. Simply by presenting a few multimodal examples within the prompt, it adapted to multiple tasks, and it achieved a new SOTA on the 9 benchmarks for which prior few-shot results were available.

Of course, performance can vary depending on the composition or order of the few-shot demonstrations, and the limitations of the underlying language model remain as they are. Nonetheless, it is important research in that it demonstrated the possibility that a single model can perform multiple vision-language tasks in a few-shot manner.

It is difficult to regard Flamingo’s few-shot ability as a result of the architecture alone. In the ablation, removing M3W dropped the overall score most sharply, from 70.7 to 53.4. Therefore, the data format itself—in which multiple images and text are interleaved like an actual prompt—can be seen to have played an important role in creating the few-shot ability.

Furthermore, the structure combining a frozen backbone with trainable connecting modules is closer to a reusable training recipe than to a single fixed model. The significance of Flamingo is also confirmed by the fact that OpenFlamingo later re-implemented this structure using open datasets and backbones.

OpenFlamingo

Introduction

OpenFlamingo is research that reproduces the previously examined Flamingo using only open data and open backbones. In that the reproduction process reveals which design factors govern in-context learning performance, it can be seen as a paper that empirically validates the “reusable recipe” perspective mentioned in Flamingo’s conclusion.

The representative form of existing vision-language models is (image, text) → text. That is, a structure that takes a single image and text as input and outputs text, with BLIP-2 as a representative example. This form naturally supports tasks such as image classification and visual question answering (VQA).

However, restricting the input to a single image greatly reduces what can be done. An autoregressive vision-language model, like the Flamingo seen earlier, takes as input a sequence in which images and text are arbitrarily interleaved (an interleaved sequence) and generates text. In this interface, demonstrations of a new task can be placed within the input sequence, and therefore few-shot in-context learning is possible without separate fine-tuning. Besides Flamingo, CM3, Kosmos-1, PALM-E, and multimodal GPT-4 belong to this family.

The problem is that most of these models are closed-source. Because their weights, training data, code, and hyperparameters are all undisclosed, it is difficult for academia to verify questions such as “how does image-text data collected from the web affect the model’s performance and safety.” There are open alternatives such as LLaVA, LLaMA-Adapter, BLIP-2, and mPLUG-Owl, but these often take only a single image as input and are trained on curated datasets such as COCO.

To fill this gap, this paper proposes OpenFlamingo, which reproduces Flamingo using only open resources, and organizes the experience gained in the process in the form of a technical report.

The structure follows exactly the design principle of Flamingo seen earlier, namely, freeze the two pretrained models and train only the Perceiver resampler and gated cross-attention modules that connect them. However, the vision encoder uses CLIP ViT-L/14 instead of NFNet-F6, and the language model uses open backbones such as MPT and RedPajama instead of Chinchilla.

Input Processing

When given an interleaved sequence, OpenFlamingo predicts the next text token conditioned on all previous text tokens and the immediately preceding image. This is the same as the per-image attention masking seen earlier in Flamingo, where the model directly cross-attends only to the immediately preceding image, while information from earlier images is conveyed indirectly through the text hidden state. Images are passed through the frozen vision encoder to extract patch features, and then passed through the trainable Perceiver resampler to be delivered to the language model’s cross-attention.

In the preprocessing stage, two special tokens are inserted.

• <image> : marks the point in the text sequence where an image is located
• <|endofchunk|> : marks the end of the text span attached to one image

For example, an input consisting of an image $x$ and the text “Hello world” is converted as follows.

<image> Hello world <|endofchunk|>

For reference, unlike Flamingo, OpenFlamingo currently does not support video input.

Model Variants

The paper releases five models that vary the language model backbone and the cross-attention insertion interval. A cross-attention interval of 4 means that a cross-attention module was inserted once every 4 layers of the language model.

Here, (Instruct) means that a backbone instruction-tuned on language-only tasks was used, not that instruction tuning was performed on vision-language tasks. Also, only the 4B models have their special-token embeddings fixed in a randomly initialized state, which was due to an implementation issue with gradient masking when using FSDP. This choice is later identified as a cause of performance degradation.

Training Dataset

Flamingo was trained on the closed datasets ALIGN (image-text pairs) and M3W (interleaved sequences). OpenFlamingo replaces each of these with an open dataset.

• LAION-2B: uses a portion of the English subset, with captions truncated to 32 tokens. Every pair in LAION-2B has a cosine similarity of at least 0.28 by the CLIP ViT-B/32 measure.

• Multimodal C4 (MMC4): unlike M3W or OBELISC, which build sequences by directly parsing HTML documents, MMC4 uses CLIP to soft-align images with the sentences within a document. For data quality, the authors remove an image from the sequence if the cosine similarity between the image and the following text is below 0.24 (by the CLIP ViT-L/14 measure), and discard the sequence itself if all images are removed. In addition, to encourage learning on multi-image sequences, they reject single-image sequences with probability 0.5.

Here an interesting trade-off appears. Raising the filtering threshold to 0.32 improves data quality, but about 58% of samples become single-image sequences and 88.7% of all sequences are discarded (versus 42.7% at 0.24). In other words, raising quality shortens the sequences, and short sequences can undermine the ability to handle many in-context examples.

Also, for the 4B models only, an additional 417K synthetic sequences generated by ChatGPT were used. The method is to have ChatGPT generate a sequence in which text and image alt-text alternate, then use that alt-text to retrieve actual images from LAION-5B to fill in.

The characteristics of each dataset are as follows.

This table becomes an important basis for interpreting the later results. MMC4 is data with long text but few images, which is an unfavorable condition for the model to learn the ability to leverage many in-context examples.

Training Procedure

• Datasets: 60 million interleaved (MMC4) examples + 120 million LAION-2B examples
• Objective: next-token prediction, with AdamW as the optimizer
• Learning rate: after an initial linear warmup, fixed at 1e-4
• Weight decay: 0.1 applied to the dense cross-attention layers
• Batch/loss: the LAION-2B batch size is twice that of the interleaved data, and the loss weights are MMC4 1 and LAION-2B 0.2 (the Flamingo defaults)
• GPU: 64 GPUs across 8 nodes on the Stability AI cluster; the 4B models use FSDP, and the rest use DDP

After the early part of training, the MMC4 loss decreases very slowly. The authors conjecture that because MMC4 sequences contain long paragraphs between images, most text tokens can be generated without referring to the image. That is, a substantial part of the loss is dominated by “how well the frozen language model predicts unrelated paragraphs.” This suggests that the loss value may not be a good indicator of multimodal ability.

Evaluation Method

Performance is measured in the 0, 4, 8, 16, and 32-shot settings on 7 vision-language datasets.

The few-shot evaluation prompt format for each task is as follows.

• Captioning: <image> Output: [caption]
• VQA: <image> Question: [question] Short answer: [answer]
• HatefulMemes: <image> is an image with: '[text]' written on it. Is it hateful? Answer: [answer]

Following Flamingo’s protocol, even in the zero-shot evaluation, two text-only examples with the image removed are placed in the prompt, and for classification tasks, prompt ensembling is applied by averaging the logits over 6 permutations of the in-context examples.

Captioning and VQA are generated with beam search of beam size 3, stopping at 20 tokens for captioning, 5 tokens for VQA, or when <|endofchunk|> is generated. HatefulMemes compares the log-likelihoods of “yes”/”no.” In-context examples are by default drawn uniformly at random from the training split, but the appendix additionally presents results using RICES (Retrieval-based In-Context Example Selection), which retrieves training examples visually similar to the test example to use as in-context examples. All evaluations are averaged over 3 seeds.

Main Results

On the average across the 7 datasets, OpenFlamingo-3B achieves about 85% of Flamingo-3B’s performance, and OpenFlamingo-9B about 89% of Flamingo-9B’s performance.

In particular, in the 0-shot and 4-shot regimes, it approaches or even surpasses Flamingo on some datasets.

On the other hand, the gap is clear on OK-VQA and TextVQA. On a 0-shot basis, OK-VQA is 6.9%p lower and TextVQA is 7.8%p lower. The authors themselves do not clearly identify why VQA performance is generally low, mentioning only that it may be related to the observations in the later discussion section.

The trend according to the number of in-context examples can be observed as follows.

The 3B and 9B models improve as the number of examples increases, but the rate of improvement is slower than Flamingo’s. As a result, the larger the number of shots, the more the gap with Flamingo-9B actually widens. The authors point to the characteristic of the pretraining data examined earlier, namely the small number of images per sequence, as the cause.

The 4B models are even more peculiar. After 4-shot or 8-shot, performance actually declines, and on several datasets they perform even worse than the smaller 3B models. This is due to the two differences of the 4B models—the RedPajama backbone and the frozen special-token embeddings—of which the latter is examined in detail in the discussion section below.

The effect of an instruction-tuned backbone can also be observed. As a result of separately training a base backbone and an instruction-tuned backbone at each scale, the instruction-tuned variant was on average better. The difference was largest for RedPajama-3B. The observation that the effect of language-only instruction tuning transfers to vision-language tasks is consistent with prior work such as Kosmos-1 and BLIP-2.

When compared with the fine-tuned SoTA, in the 32-shot RICES setting OpenFlamingo-9B shows about 62% of the performance on average, which is even lower than Flamingo-9B’s 72%. In other words, a substantial gap still exists between the few-shot approach and task-specific fine-tuning.

Looking at the performance on each dataset when the RICES method is applied and the difference from the Random method:

In most settings RICES greatly boosts performance, but on Flickr-30K it actually drops sharply. The authors’ interpretation is that when the retrieved examples are too similar, the model copies the example captions verbatim. Indeed, in the paper’s example, for a test image of a white dog holding a yellow toy, the model patches together phrases from the example captions to generate the incorrect description “a yellow dog holding a green toy.” This is a case showing that in-context learning can degenerate into surface-level copying beyond learning the task format.

Discussion

1. The effect of frozen embeddings

To verify the anomaly of the 4B models, the authors train a small model based on OPT-125M on up to 20 million interleaved samples, changing only whether the <image> and <|endofchunk|> embeddings are trained.

Special-token embeddings fixed in a randomly initialized state caused a performance drop of 4.6 CIDEr on COCO and 12.1%p on VQAv2. The fact that just a few token embeddings marking the sequence structure can greatly change downstream performance shows the impact that implementation details have on the results.

2. The effect of the language model backbone

Training for a long time did not greatly raise VQAv2 performance. In contrast, changing the language model backbone made a large difference.

Merely switching from OPT-1.3B to MPT-1B (Instruct) raised 0-shot performance by about 10%p. In other words, one can interpret that a substantial part of VQA performance depends on the quality of the language model itself rather than the amount of multimodal training.

3. Common VQA failure types

In particular, the verbosity problem is a case where the model’s actual understanding diverges from the formal requirements of the evaluation metric, a point to keep in mind when interpreting performance numbers.

Conclusion

Rather than proposing a new methodology, this paper is a technical report that documents the entire process of reproducing a closed-source multimodal model (Flamingo) with open resources. The core content can be summarized in the following three points.

• Substantial reproduction is possible with open resources alone: using only CLIP, open-source language models, LAION-2B, and MMC4, it reached about 80-89% of Flamingo’s performance.
• The gap appears mainly in in-context learning ability: it approaches Flamingo at 0-shot and 4-shot, but the gap widens as the number of shots increases, which is connected to the characteristic of the pretraining data having few images per sequence.
• Design details greatly change the results: whether the special-token embeddings were frozen, which language model was used as the backbone, and how the in-context examples were selected all had a large impact on performance.

The significance of this research lies not in achieving the best performance, but in bringing into the open a training process that until then could only be verified within a few institutions. By releasing all of the weights, code, hyperparameters, and evaluation suite, it enabled academia to directly verify how web-scraped data affects a model’s capabilities and risks, and it reports even failure cases such as the 4B models’ performance degradation and the stagnant MMC4 loss as they are, providing practical information for follow-up research. Indeed, Otter, fine-tuned with MIMIC-IT, and Multimodal-GPT, trained on vision-language instruction data, were developed on top of OpenFlamingo.

In particular, the trade-off surrounding MMC4’s filtering threshold is striking. Raising the threshold improves image-text alignment, but shortens the sequences and discards most of the data, which can harm the ability to handle many examples. This suggests that data quality and data structure are separate axes, and that in multimodal in-context learning the latter may be especially important.

Of course, the limitations are also clear. Performance at 80-89% of Flamingo can hardly be regarded as a complete reproduction, and especially since the gap widens as the number of shots increases, in-context learning ability itself was not fully replicated. The authors estimate the cause to be the image density of the pretraining data, but did not verify this with a directly controlled experiment, and it should also be noted that the comparison with Flamingo is not a result re-run in the same evaluation pipeline but a citation of the numbers reported in Flamingo’s paper. In addition, the model was trained on web-scraped data and did not undergo safety-focused fine-tuning, so it inherits the risks of the underlying language model as they are, and it does not support video input, so it does not fully cover Flamingo’s range of functionality. Even so, the very fact that it explicitly states these limitations itself and openly shares the entire reproduction process is precisely the way this paper contributes to the community.

Closing Remarks

Placing the two papers side by side, it becomes clear that what Flamingo presented was not a specific model so much as a reusable recipe of a frozen backbone + trainable connecting modules. OpenFlamingo showed that this recipe works to a substantial degree even with open resources alone, but at the same time showed that the recipe alone is not enough. Just as removing M3W produced the largest performance drop in Flamingo’s ablation, in OpenFlamingo too, pretraining data with few images per sequence was identified as the bottleneck for the ability to leverage many shots. In the end, the consistent message running through both papers is that the core of multimodal few-shot ability lies, together with the architecture, in the structure of data in which images and text are naturally interleaved.