Towards Transferable Adversarial Attacks on Vision Transformers

Vision transformers (ViTs) have demonstrated impressive performance on a series of computer vision tasks, yet they still suffer from adversarial examples. In this paper, we posit that adversarial attacks on transformers should be specially tailored for their architecture, jointly considering both patches and self-attention, in order to achieve high transferability. More specifically, we introduce a dual attack framework, which contains a Pay No Attention (PNA) attack and a PatchOut attack, to improve the transferability of adversarial samples across different ViTs. We show that skipping the gradients of attention during backpropagation can generate adversarial examples with high transferability. In addition, adversarial perturbations generated by optimizing randomly sampled subsets of patches at each iteration achieve higher attack success rates than attacks using all patches. We evaluate the transferability of attacks on state-of-the-art ViTs, CNNs and robustly trained CNNs. The results of these experiments demonstrate that the proposed dual attack can greatly boost transferability between ViTs and from ViTs to CNNs. In addition, the proposed method can easily be combined with existing transfer methods to boost performance.

Authors

• Zhipeng Wei Shanghai Key Lab of Intell. Info. Processing, School of Computer Science, Fudan University, Shanghai Collaborative Innovation Center on Intelligent Visual Computing
• Jingjing Chen Shanghai Key Lab of Intell. Info. Processing, School of Computer Science, Fudan University, Shanghai Collaborative Innovation Center on Intelligent Visual Computing
• Micah Goldblum
• Zuxuan Wu Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University Shanghai Collaborative Innovation Center of Intelligent Visual Computing
• Tom Goldstein University of Maryland, College Park
• Yu-Gang Jiang Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University Shanghai Collaborative Innovation Center of Intelligent Visual Computing

Keywords

No keywords are indexed for this paper.