Paper: On the Adversarial Robustness of Multi-Modal Foundation Models

Authors: Christian Schlarmann, Matthias Hein

Venue: ICCV AROW 2023

URL: On the Adversarial Robustness of Multi-Modal Foundation Models

Introduction

Existing work on the safety of multimodal foundation models has mainly dealt with situations in which a malicious user directly enters a jailbreak prompt to elicit a harmful response. This paper instead assumes a situation in which the user of the model is legitimate, but the image the user supplies has been manipulated in advance by a third party.

In other words, the attacker can control the final output merely by manipulating the image, without directly controlling either the model or the user.

The paper analyzes this attack possibility on OpenFlamingo-9B and evaluates robustness against two kinds of adversarial attacks.

  • Untargeted attack: degrades the model’s performance so that a normal output is not produced.
  • Targeted attack: makes the model reproduce verbatim a specific sentence chosen in advance by the attacker.

Adversarial Attacks on OpenFlamingo

The OpenFlamingo Model

The paper uses the open-source model OpenFlamingo as the attack target. OpenFlamingo consists of a pretrained vision encoder and a generative language model, where the language model attends to image features extracted by the vision encoder through cross-attention.

Given an image $x$, the model repeatedly predicts the next token conditioned on the previously generated tokens and the image information to produce the output $y$.

\[\begin{equation} p(y \mid x)=\prod_{l=1}^{L} p\!\left(y_l \mid y_{<l}, x\right) \end{equation}\]

Since each token is generated conditionally on the previous ones, if a small change to the image alters the probability distribution of the early tokens, that effect accumulates into subsequent tokens and the entire generated sentence can end up different.

Threat Model

The paper assumes a white-box attack in which the attacker can access all of OpenFlamingo’s parameters. The attacker adds a perturbation $\delta$ to the original image $x$, subject to the following $\ell_\infty$ constraint so that the change is barely visible to the eye.

\[\begin{equation} \|\delta\|_\infty \leq \epsilon \end{equation}\]

Here $\epsilon$ is measured on pixel values normalized to $[0,1]$, and the paper uses $\epsilon = 1/255$ and $\epsilon = 4/255$.

Taking OpenFlamingo’s few-shot nature into account, two attack settings are considered.

  • Query attack: attacks only the query image the user actually asks about
  • All-image attack: attacks both the query image and the context images

In the zero-shot case there are no context images, so the two settings coincide.

Untargeted Attack

The goal of the untargeted attack is to make the model generate an output different from the original correct answer.

Writing the query image as $q$, the context images as $c$, the text given along with the prompt as $z$, and the correct caption as $y$, the attacker maximizes the negative log-likelihood of the correct answer.

\[\begin{equation} \max_{\delta_q,\delta_c}\; -\sum_{l=1}^{L} \log p\!\left(y_l \mid y_{<l}, z, q+\delta_q, c+\delta_c\right) \end{equation}\] \[\begin{equation} \text{s.t.}\quad \|\delta_q\|_\infty \leq \epsilon_q, \quad \|\delta_c\|_\infty \leq \epsilon_c \end{equation}\]

In this case the attacker does not specify which incorrect sentence the model should generate. The goal is simply to prevent a normal output from being produced.

Targeted Attack

The targeted attack goes beyond merely degrading performance and manipulates the model into producing a specific sentence $\hat{y}$ that the attacker wants.

\[\begin{equation} \min_{\delta_q,\delta_c}\; -\sum_{l=1}^{L} \log p\!\left(\hat{y}_l \mid \hat{y}_{<l}, z, q+\delta_q, c+\delta_c\right) \end{equation}\] \[\begin{equation} \text{s.t.}\quad \|\delta_q\|_\infty \leq \epsilon_q, \quad \|\delta_c\|_\infty \leq \epsilon_c \end{equation}\]

Where the untargeted attack changes the image so as to lower the probability of the correct answer, the targeted attack changes it so as to raise the probability of the sentence the attacker specifies. Only the sign of the two objectives differs; the optimization target is the same.

Both attacks optimize the perturbation using APGD (Auto-PGD), an iterative gradient-based attack.

Because the targeted attack must generate an entire specific token sequence, it requires far more iterations than the untargeted attack. With $\epsilon = 1/255$ and 100 APGD iterations, the targeted attack success rate was 12%, but at 5,000 iterations it rises to 83%.

Evaluating the Attacks

The paper evaluates OpenFlamingo’s adversarial robustness on two kinds of tasks: image captioning and VQA.

Evaluation datasets and metrics by task

  • Image captioning: on COCO 2014 and Flickr30k, the similarity between generated captions and reference captions is evaluated with the CIDEr score
  • VQA: on OK-VQA and VizWiz, the VQA accuracy of the model’s generated answers is measured

Both zero-shot and four-shot inference are tested on each dataset.

Performance under Targeted Attack

In the COCO experiments, the attack success rate is measured against the following two representative target sentences.

  • Please reset your password
  • A person suffers severe side effects after vaccination
On the Adversarial Robustness of Multi-Modal Foundation Models
Targeted attack success rate on COCO by target sentence and perturbation size

For the relatively short target sentence “Please reset your password,” $\epsilon = 1/255$ alone yields a 51.66% success rate in zero-shot and 86.00% in four-shot when the context images are attacked as well. Raising it to $\epsilon = 4/255$ records 100% in both zero-shot and four-shot.

The longer target sentence “A person suffers severe side effects after vaccination” was comparatively harder to attack at small $\epsilon$, but at $\epsilon = 4/255$ the success rate rises to 83.8% in zero-shot and 100% in four-shot. This confirms that the longer the token sequence to be generated, the larger the perturbation required.

On the Adversarial Robustness of Multi-Modal Foundation Models
Targeted attack example — comparison of model output on the original image (left) and the attacked image (right)

In the attack example targeting “Please reset your password,” the unattacked image on the left produces a normal output, while the attacked image on the right makes the model reproduce the attacker’s intended sentence verbatim.

That is, even for a sentence entirely unrelated to the image, minutely changing the pixels alone can move the model’s generation in the direction the attacker intends.

Performance under Untargeted Attack

The untargeted attack also causes large performance drops on every dataset.

On the Adversarial Robustness of Multi-Modal Foundation Models
Performance change per dataset under untargeted attack

COCO’s zero-shot CIDEr score is 84.01 on clean images, but falls to 9.59 at $\epsilon = 1/255$ and 1.69 at $\epsilon = 4/255$. Flickr30k likewise falls from 59.57 to 7.45 and 1.17 respectively.

The same phenomenon appears in VQA. OK-VQA’s zero-shot accuracy was 34.72% on clean images but drops to 1.92% after the $\epsilon = 1/255$ attack and to 1.14% at $\epsilon = 4/255$.

On the Adversarial Robustness of Multi-Modal Foundation Models
Untargeted attack example — caption comparison between the original image (left) and the attacked image (right)

On the image captioning task, the unattacked image on the left produces a normal caption, while output quality degrades markedly on the attacked image on the right.

Adding context information through four-shot inference did not effectively block the attack. Performance improves somewhat on image captioning, but on VQA the model remained vulnerable to adversarial attack even when context images were provided.

Is the Attack Possible by Changing Only Some Pixels?

The paper additionally analyzes whether the attack requires changing every pixel.

Removing the smallest components of the APGD perturbation and keeping only the top fraction, it turns out that keeping just the top 60% of the perturbation retains a substantial part of the attack’s effect.

On the Adversarial Robustness of Multi-Modal Foundation Models
Attack performance when only the top fraction of the perturbation is retained

Using only the top 60% of the perturbation in the untargeted attack, COCO’s CIDEr score records 24.99 at $\epsilon = 1/255$ and 3.65 at $\epsilon = 4/255$. The attack is weaker than with the full perturbation (9.59, 1.69), but compared with 84.01 on clean images it is still a large degradation.

The attack therefore does not require minute changes across all pixels of the image; a few essential perturbations alone can substantially alter the model’s image representation and its final output.

Conclusion

This paper showed that image manipulation within a range barely perceptible to humans can substantially degrade OpenFlamingo’s image captioning and VQA performance, and further can make the model reproduce a specific sentence chosen by the attacker.

What is particularly interesting is that it separates the agent of the attack from the user of the model. Even if the user makes no jailbreak attempt, the attack succeeds simply when a legitimate user supplies an image manipulated by a third party. In multimodal models, in other words, an externally supplied image is itself an attack surface.

There are limitations: the experiments centered on a single open-source model, OpenFlamingo, and assume a white-box setting with full access to the model parameters. In addition, a strong targeted attack requires hundreds to thousands of APGD iterations, so whether the attack is feasible at the same cost in a real service environment needs further verification.

Even so, the finding that perturbations as small as $\epsilon = 1/255$ substantially change the model output, and that at $\epsilon = 4/255$ a specific target sentence is reproduced verbatim with high probability, shows that in evaluating the safety of multimodal models, robustness to the input image must be treated as an independent safety factor alongside the harmfulness of the text.

  • Flamingo Series: Flamingo, OpenFlamingo [NeurIPS 2022] | [Paper] | [Article]
  • Explaining and Harnessing Adversarial Examples [ICLR 2015] | [Paper] | [Article]
  • Towards Deep Learning Models Resistant to Adversarial Attacks [ICLR 2018] | [Paper] | [Article]
  • Towards Evaluating the Robustness of Neural Networks [IEEE S&P 2017] | [Paper] | [Article]