http://arxiv.org/abs/2501.13563
Black-Box Adversarial Attack on Vision Language Models for Autonomous Driving. (99%)
Lu Wang; Tianyuan Zhang; Yang Qu; Siyuan Liang; Yuwei Chen; Aishan Liu; Xianglong Liu; Dacheng Tao
  Vision-language models (VLMs) have significantly advanced autonomous driving
(AD) by enhancing reasoning capabilities; however, these models remain highly
susceptible to adversarial attacks. While existing research has explored
white-box attacks to some extent, the more practical and challenging black-box
scenarios remain largely underexplored due to their inherent difficulty. In
this paper, we take the first step toward designing black-box adversarial
attacks specifically targeting VLMs in AD. We identify two key challenges for
achieving effective black-box attacks in this context: the effectiveness across
driving reasoning chains in AD systems and the dynamic nature of driving
scenarios. To address this, we propose Cascading Adversarial Disruption (CAD).
It first introduces Decision Chain Disruption, which targets low-level
reasoning breakdown by generating and injecting deceptive semantics, ensuring
the perturbations remain effective across the entire decision-making chain.
Building on this, we present Risky Scene Induction, which addresses dynamic
adaptation by leveraging a surrogate VLM to understand and construct high-level
risky scenarios that are likely to result in critical errors in the current
driving contexts. Extensive experiments conducted on multiple AD VLMs and
benchmarks demonstrate that CAD achieves state-of-the-art attack effectiveness,
significantly outperforming existing methods (+13.43% on average). Moreover, we
validate its practical applicability through real-world attacks on AD vehicles
powered by VLMs, where the route completion rate drops by 61.11% and the
vehicle crashes directly into the obstacle vehicle with adversarial patches.
Finally, we release CADA dataset, comprising 18,808 adversarial
visual-question-answer pairs, to facilitate further evaluation and research in
this critical domain. Our codes and dataset will be available after paper's
acceptance.


http://arxiv.org/abs/2501.13782
Defending against Adversarial Malware Attacks on ML-based Android Malware Detection Systems. (98%)
Ping He; Lorenzo Cavallaro; Shouling Ji
  Android malware presents a persistent threat to users' privacy and data
integrity. To combat this, researchers have proposed machine learning-based
(ML-based) Android malware detection (AMD) systems. However, adversarial
Android malware attacks compromise the detection integrity of the ML-based AMD
systems, raising significant concerns. Existing defenses against adversarial
Android malware provide protections against feature space attacks which
generate adversarial feature vectors only, leaving protection against realistic
threats from problem space attacks which generate real adversarial malware an
open problem. In this paper, we address this gap by proposing ADD, a practical
adversarial Android malware defense framework designed as a plug-in to enhance
the adversarial robustness of the ML-based AMD systems against problem space
attacks. Our extensive evaluation across various ML-based AMD systems
demonstrates that ADD is effective against state-of-the-art problem space
adversarial Android malware attacks. Additionally, ADD shows the defense
effectiveness in enhancing the adversarial robustness of real-world antivirus
solutions.


http://arxiv.org/abs/2501.13776
Crossfire: An Elastic Defense Framework for Graph Neural Networks Under Bit Flip Attacks. (83%)
Lorenz Kummer; Samir Moustafa; Wilfried Gansterer; Nils Kriege
  Bit Flip Attacks (BFAs) are a well-established class of adversarial attacks,
originally developed for Convolutional Neural Networks within the computer
vision domain. Most recently, these attacks have been extended to target Graph
Neural Networks (GNNs), revealing significant vulnerabilities. This new
development naturally raises questions about the best strategies to defend GNNs
against BFAs, a challenge for which no solutions currently exist. Given the
applications of GNNs in critical fields, any defense mechanism must not only
maintain network performance, but also verifiably restore the network to its
pre-attack state. Verifiably restoring the network to its pre-attack state also
eliminates the need for costly evaluations on test data to ensure network
quality. We offer first insights into the effectiveness of existing honeypot-
and hashing-based defenses against BFAs adapted from the computer vision domain
to GNNs, and characterize the shortcomings of these approaches. To overcome
their limitations, we propose Crossfire, a hybrid approach that exploits weight
sparsity and combines hashing and honeypots with bit-level correction of
out-of-distribution weight elements to restore network integrity. Crossfire is
retraining-free and does not require labeled data. Averaged over 2,160
experiments on six benchmark datasets, Crossfire offers a 21.8% higher
probability than its competitors of reconstructing a GNN attacked by a BFA to
its pre-attack state. These experiments cover up to 55 bit flips from various
attacks. Moreover, it improves post-repair prediction quality by 10.85%.
Computational and storage overheads are negligible compared to the inherent
complexity of even the simplest GNNs.


http://arxiv.org/abs/2501.13676
Certified Robustness Under Bounded Levenshtein Distance. (67%)
Elias Abad Rocamora; Grigorios G. Chrysos; Volkan Cevher
  Text classifiers suffer from small perturbations, that if chosen
adversarially, can dramatically change the output of the model. Verification
methods can provide robustness certificates against such adversarial
perturbations, by computing a sound lower bound on the robust accuracy.
Nevertheless, existing verification methods incur in prohibitive costs and
cannot practically handle Levenshtein distance constraints. We propose the
first method for computing the Lipschitz constant of convolutional classifiers
with respect to the Levenshtein distance. We use these Lipschitz constant
estimates for training 1-Lipschitz classifiers. This enables computing the
certified radius of a classifier in a single forward pass. Our method, LipsLev,
is able to obtain $38.80$% and $13.93$% verified accuracy at distance $1$ and
$2$ respectively in the AG-News dataset, while being $4$ orders of magnitude
faster than existing approaches. We believe our work can open the door to more
efficient verification in the text domain.


http://arxiv.org/abs/2501.13894
Logical Maneuvers: Detecting and Mitigating Adversarial Hardware Faults in Space. (1%)
Fatemeh Khojasteh Dana; Saleh Khalaj Monfared; Shahin Tajik
  Satellites are highly vulnerable to adversarial glitches or high-energy
radiation in space, which could cause faults on the onboard computer. Various
radiation- and fault-tolerant methods, such as error correction codes (ECC) and
redundancy-based approaches, have been explored over the last decades to
mitigate temporary soft errors on software and hardware. However, conventional
ECC methods fail to deal with hard errors or permanent faults in the hardware
components. This work introduces a detection- and response-based countermeasure
to deal with partially damaged processor chips. It recovers the processor chip
from permanent faults and enables continuous operation with available undamaged
resources on the chip. We incorporate digitally-compatible delay-based sensors
on the target processor's chip to reliably detect the incoming radiation or
glitching attempts on the physical fabric of the chip, even before a fault
occurs. Upon detecting a fault in one or more components of the processor's
arithmetic logic unit (ALU), our countermeasure employs adaptive software
recompilations to resynthesize and substitute the affected instructions with
instructions of still functioning components to accomplish the task.
Furthermore, if the fault is more widespread and prevents the correct operation
of the entire processor, our approach deploys adaptive hardware partial
reconfigurations to replace and reroute the failed components to undamaged
locations of the chip. To validate our claims, we deploy a high-energy
near-infrared (NIR) laser beam on a RISC-V processor implemented on a 28~nm
FPGA to emulate radiation and even hard errors by partially damaging the FPGA
fabric. We demonstrate that our sensor can confidently detect the radiation and
trigger the processor testing and fault recovery mechanisms. Finally, we
discuss the overhead imposed by our countermeasure.


http://arxiv.org/abs/2501.13336
Gradient-Free Adversarial Purification with Diffusion Models. (99%)
Xuelong Dai; Dong Wang; Duan Mingxing; Bin Xiao
  Adversarial training and adversarial purification are two effective and
practical defense methods to enhance a model's robustness against adversarial
attacks. However, adversarial training necessitates additional training, while
adversarial purification suffers from low time efficiency. More critically,
current defenses are designed under the perturbation-based adversarial threat
model, which is ineffective against the recently proposed unrestricted
adversarial attacks. In this paper, we propose an effective and efficient
adversarial defense method that counters both perturbation-based and
unrestricted adversarial attacks. Our defense is inspired by the observation
that adversarial attacks are typically located near the decision boundary and
are sensitive to pixel changes. To address this, we introduce adversarial
anti-aliasing to mitigate adversarial modifications. Additionally, we propose
adversarial super-resolution, which leverages prior knowledge from clean
datasets to benignly recover images. These approaches do not require additional
training and are computationally efficient without calculating gradients.
Extensive experiments against both perturbation-based and unrestricted
adversarial attacks demonstrate that our defense method outperforms
state-of-the-art adversarial purification methods.


http://arxiv.org/abs/2501.12761
Modality Unified Attack for Omni-Modality Person Re-Identification. (98%)
Yuan Bian; Min Liu; Yunqi Yi; Xueping Wang; Yunfeng Ma; Yaonan Wang
  Deep learning based person re-identification (re-id) models have been widely
employed in surveillance systems. Recent studies have demonstrated that
black-box single-modality and cross-modality re-id models are vulnerable to
adversarial examples (AEs), leaving the robustness of multi-modality re-id
models unexplored. Due to the lack of knowledge about the specific type of
model deployed in the target black-box surveillance system, we aim to generate
modality unified AEs for omni-modality (single-, cross- and multi-modality)
re-id models. Specifically, we propose a novel Modality Unified Attack method
to train modality-specific adversarial generators to generate AEs that
effectively attack different omni-modality models. A multi-modality model is
adopted as the surrogate model, wherein the features of each modality are
perturbed by metric disruption loss before fusion. To collapse the common
features of omni-modality models, Cross Modality Simulated Disruption approach
is introduced to mimic the cross-modality feature embeddings by intentionally
feeding images to non-corresponding modality-specific subnetworks of the
surrogate model. Moreover, Multi Modality Collaborative Disruption strategy is
devised to facilitate the attacker to comprehensively corrupt the informative
content of person images by leveraging a multi modality feature collaborative
metric disruption loss. Extensive experiments show that our MUA method can
effectively attack the omni-modality re-id models, achieving 55.9%, 24.4%,
49.0% and 62.7% mean mAP Drop Rate, respectively.


http://arxiv.org/abs/2501.13094
Robust Representation Consistency Model via Contrastive Denoising. (84%)
Jiachen Lei; Julius Berner; Jiongxiao Wang; Zhongzhu Chen; Zhongjia Ba; Kui Ren; Jun Zhu; Anima Anandkumar
  Robustness is essential for deep neural networks, especially in
security-sensitive applications. To this end, randomized smoothing provides
theoretical guarantees for certifying robustness against adversarial
perturbations. Recently, diffusion models have been successfully employed for
randomized smoothing to purify noise-perturbed samples before making
predictions with a standard classifier. While these methods excel at small
perturbation radii, they struggle with larger perturbations and incur a
significant computational overhead during inference compared to classical
methods. To address this, we reformulate the generative modeling task along the
diffusion trajectories in pixel space as a discriminative task in the latent
space. Specifically, we use instance discrimination to achieve consistent
representations along the trajectories by aligning temporally adjacent points.
After fine-tuning based on the learned representations, our model enables
implicit denoising-then-classification via a single prediction, substantially
reducing inference costs. We conduct extensive experiments on various datasets
and achieve state-of-the-art performance with minimal computation budget during
inference. For example, our method outperforms the certified accuracy of
diffusion-based methods on ImageNet across all perturbation radii by 5.3% on
average, with up to 11.6% at larger radii, while reducing inference costs by
85$\times$ on average. Codes are available at:
https://github.com/jiachenlei/rRCM.


http://arxiv.org/abs/2501.13302
Watching the AI Watchdogs: A Fairness and Robustness Analysis of AI Safety Moderation Classifiers. (11%)
Akshit Achara; Anshuman Chhabra
  AI Safety Moderation (ASM) classifiers are designed to moderate content on
social media platforms and to serve as guardrails that prevent Large Language
Models (LLMs) from being fine-tuned on unsafe inputs. Owing to their potential
for disparate impact, it is crucial to ensure that these classifiers: (1) do
not unfairly classify content belonging to users from minority groups as unsafe
compared to those from majority groups and (2) that their behavior remains
robust and consistent across similar inputs. In this work, we thus examine the
fairness and robustness of four widely-used, closed-source ASM classifiers:
OpenAI Moderation API, Perspective API, Google Cloud Natural Language (GCNL)
API, and Clarifai API. We assess fairness using metrics such as demographic
parity and conditional statistical parity, comparing their performance against
ASM models and a fair-only baseline. Additionally, we analyze robustness by
testing the classifiers' sensitivity to small and natural input perturbations.
Our findings reveal potential fairness and robustness gaps, highlighting the
need to mitigate these issues in future versions of these models.


http://arxiv.org/abs/2501.12736
Bad-PFL: Exploring Backdoor Attacks against Personalized Federated Learning. (9%)
Mingyuan Fan; Zhanyi Hu; Fuyi Wang; Cen Chen
  Data heterogeneity and backdoor attacks rank among the most significant
challenges facing federated learning (FL). For data heterogeneity, personalized
federated learning (PFL) enables each client to maintain a private personalized
model to cater to client-specific knowledge. Meanwhile, vanilla FL has proven
vulnerable to backdoor attacks. However, recent advancements in PFL community
have demonstrated a potential immunity against such attacks. This paper
explores this intersection further, revealing that existing federated backdoor
attacks fail in PFL because backdoors about manually designed triggers struggle
to survive in personalized models. To tackle this, we design Bad-PFL, which
employs features from natural data as our trigger. As long as the model is
trained on natural data, it inevitably embeds the backdoor associated with our
trigger, ensuring its longevity in personalized models. Moreover, our trigger
undergoes mutual reinforcement training with the model, further solidifying the
backdoor's durability and enhancing attack effectiveness. The large-scale
experiments across three benchmark datasets demonstrate the superior
performance of our attack against various PFL methods, even when equipped with
state-of-the-art defense mechanisms.


http://arxiv.org/abs/2501.13291
Are We Learning the Right Features?A Framework for Evaluating DL-Based Software Vulnerability Detection Solutions. (2%)
Satyaki Das; Syeda Tasnim Fabiha; Saad Shafiq; Nenad Medvidovic
  Recent research has revealed that the reported results of an emerging body of
DL-based techniques for detecting software vulnerabilities are not
reproducible, either across different datasets or on unseen samples. This paper
aims to provide the foundation for properly evaluating the research in this
domain. We do so by analyzing prior work and existing vulnerability datasets
for the syntactic and semantic features of code that contribute to
vulnerability, as well as features that falsely correlate with vulnerability.
We provide a novel, uniform representation to capture both sets of features,
and use this representation to detect the presence of both vulnerability and
spurious features in code. To this end, we design two types of code
perturbations: feature preserving perturbations (FPP) ensure that the
vulnerability feature remains in a given code sample, while feature eliminating
perturbations (FEP) eliminate the feature from the code sample. These
perturbations aim to measure the influence of spurious and vulnerability
features on the predictions of a given vulnerability detection solution. To
evaluate how the two classes of perturbations influence predictions, we
conducted a large-scale empirical study on five state-of-the-art DL-based
vulnerability detectors. Our study shows that, for vulnerability features, only
~2% of FPPs yield the undesirable effect of a prediction changing among the
five detectors on average. However, on average, ~84% of FEPs yield the
undesirable effect of retaining the vulnerability predictions. For spurious
features, we observed that FPPs yielded a drop in recall up to 29% for
graph-based detectors. We present the reasons underlying these results and
suggest strategies for improving DNN-based vulnerability detectors. We provide
our perturbation-based evaluation framework as a public resource to enable
independent future evaluation of vulnerability detectors.


http://arxiv.org/abs/2501.11901
Enhancing Adversarial Transferability via Component-Wise Augmentation Method. (99%)
Hangyu Liu; Bo Peng; Pengxiang Ding; Donglin Wang
  Deep Neural Networks (DNNs) are highly vulnerable to adversarial examples,
which pose significant challenges in security-sensitive applications. Among
various adversarial attack strategies, input transformation-based attacks have
demonstrated remarkable effectiveness in enhancing adversarial transferability.
However, existing methods fail to diversify attention regions across models
adequately and introduce excessive information loss during transformations. In
this paper, we introduce a novel input transformation-based method, termed
Component-Wise Augmentation (CWA), designed to enhance transferability by
locally applying block-wise transformations. CWA strategically integrates
interpolation and selective rotation on individual image blocks to diversify
model attention regions while preserving semantic integrity. Extensive
experiments on the standard ImageNet dataset show that CWA consistently
outperforms state-of-the-art methods in both attack success rates and stability
across CNN- and Transformer-based models, while also demonstrating superior
performance against multiple defense methods.


http://arxiv.org/abs/2501.12275
With Great Backbones Comes Great Adversarial Transferability. (98%)
Erik Arakelyan; Karen Hambardzumyan; Davit Papikyan; Pasquale Minervini; Albert Gordo; Isabelle Augenstein; Aram H. Markosyan
  Advances in self-supervised learning (SSL) for machine vision have improved
representation robustness and model performance, giving rise to pre-trained
backbones like \emph{ResNet} and \emph{ViT} models tuned with SSL methods such
as \emph{SimCLR}. Due to the computational and data demands of pre-training,
the utilization of such backbones becomes a strenuous necessity. However,
employing these backbones may inherit vulnerabilities to adversarial attacks.
While adversarial robustness has been studied under \emph{white-box} and
\emph{black-box} settings, the robustness of models tuned on pre-trained
backbones remains largely unexplored. Additionally, the role of tuning
meta-information in mitigating exploitation risks is unclear. This work
systematically evaluates the adversarial robustness of such models across
$20,000$ combinations of tuning meta-information, including fine-tuning
techniques, backbone families, datasets, and attack types. We propose using
proxy models to transfer attacks, simulating varying levels of target knowledge
by fine-tuning these proxies with diverse configurations. Our findings reveal
that proxy-based attacks approach the effectiveness of \emph{white-box}
methods, even with minimal tuning knowledge. We also introduce a naive
"backbone attack," leveraging only the backbone to generate adversarial
samples, which outperforms \emph{black-box} attacks and rivals \emph{white-box}
methods, highlighting critical risks in model-sharing practices. Finally, our
ablations reveal how increasing tuning meta-information impacts attack
transferability, measuring each meta-information combination.


http://arxiv.org/abs/2501.11902
Transferable Adversarial Attacks on Audio Deepfake Detection. (98%)
Muhammad Umar Farooq; Awais Khan; Kutub Uddin; Khalid Mahmood Malik
  Audio deepfakes pose significant threats, including impersonation, fraud, and
reputation damage. To address these risks, audio deepfake detection (ADD)
techniques have been developed, demonstrating success on benchmarks like
ASVspoof2019. However, their resilience against transferable adversarial
attacks remains largely unexplored. In this paper, we introduce a transferable
GAN-based adversarial attack framework to evaluate the effectiveness of
state-of-the-art (SOTA) ADD systems. By leveraging an ensemble of surrogate ADD
models and a discriminator, the proposed approach generates transferable
adversarial attacks that better reflect real-world scenarios. Unlike previous
methods, the proposed framework incorporates a self-supervised audio model to
ensure transcription and perceptual integrity, resulting in high-quality
adversarial attacks. Experimental results on benchmark dataset reveal that SOTA
ADD systems exhibit significant vulnerabilities, with accuracies dropping from
98% to 26%, 92% to 54%, and 94% to 84% in white-box, gray-box, and black-box
scenarios, respectively. When tested in other data sets, performance drops of
91% to 46%, and 94% to 67% were observed against the In-the-Wild and WaveFake
data sets, respectively. These results highlight the significant
vulnerabilities of existing ADD systems and emphasize the need to enhance their
robustness against advanced adversarial threats to ensure security and
reliability.


http://arxiv.org/abs/2501.12516
Robustness of Selected Learning Models under Label-Flipping Attack. (82%)
Sarvagya Bhargava; Mark Stamp
  In this paper we compare traditional machine learning and deep learning
models trained on a malware dataset when subjected to adversarial attack based
on label-flipping. Specifically, we investigate the robustness of Support
Vector Machines (SVM), Random Forest, Gaussian Naive Bayes (GNB), Gradient
Boosting Machine (GBM), LightGBM, XGBoost, Multilayer Perceptron (MLP),
Convolutional Neural Network (CNN), MobileNet, and DenseNet models when facing
varying percentages of misleading labels. We empirically assess the the
accuracy of each of these models under such an adversarial attack on the
training data. This research aims to provide insights into which models are
inherently more robust, in the sense of being better able to resist intentional
disruptions to the training data. We find wide variation in the robustness of
the models tested to adversarial attack, with our MLP model achieving the best
combination of initial accuracy and robustness.


http://arxiv.org/abs/2501.12183
Extend Adversarial Policy Against Neural Machine Translation via Unknown Token. (80%)
Wei Zou; Shujian Huang; Jiajun Chen
  Generating adversarial examples contributes to mainstream neural machine
translation~(NMT) robustness. However, popular adversarial policies are apt for
fixed tokenization, hindering its efficacy for common character perturbations
involving versatile tokenization. Based on existing adversarial generation via
reinforcement learning~(RL), we propose the `DexChar policy' that introduces
character perturbations for the existing mainstream adversarial policy based on
token substitution. Furthermore, we improve the self-supervised matching that
provides feedback in RL to cater to the semantic constraints required during
training adversaries. Experiments show that our method is compatible with the
scenario where baseline adversaries fail, and can generate high-efficiency
adversarial examples for analysis and optimization of the system.


http://arxiv.org/abs/2501.12522
Topology of Out-of-Distribution Examples in Deep Neural Networks. (13%)
Esha Datta; Johanna Hennig; Eva Domschot; Connor Mattes; Michael R. Smith
  As deep neural networks (DNNs) become increasingly common, concerns about
their robustness do as well. A longstanding problem for deployed DNNs is their
behavior in the face of unfamiliar inputs; specifically, these models tend to
be overconfident and incorrect when encountering out-of-distribution (OOD)
examples. In this work, we present a topological approach to characterizing OOD
examples using latent layer embeddings from DNNs. Our goal is to identify
topological features, referred to as landmarks, that indicate OOD examples. We
conduct extensive experiments on benchmark datasets and a realistic DNN model,
revealing a key insight for OOD detection. Well-trained DNNs have been shown to
induce a topological simplification on training data for simple models and
datasets; we show that this property holds for realistic, large-scale test and
training data, but does not hold for OOD examples. More specifically, we find
that the average lifetime (or persistence) of OOD examples is statistically
longer than that of training or test examples. This indicates that DNNs
struggle to induce topological simplification on unfamiliar inputs. Our
empirical results provide novel evidence of topological simplification in
realistic DNNs and lay the groundwork for topologically-informed OOD detection
strategies.


http://arxiv.org/abs/2501.12123
FedCLEAN: byzantine defense by CLustering Errors of Activation maps in Non-IID federated learning environments. (11%)
Mehdi Ben Ghali; Reda Bellafqira; Gouenou Coatrieux
  Federated Learning (FL) enables clients to collaboratively train a global
model using their local datasets while reinforcing data privacy. However, FL is
susceptible to poisoning attacks. Existing defense mechanisms assume that
clients' data are independent and identically distributed (IID), making them
ineffective in real-world applications where data are non-IID. This paper
presents FedCLEAN, the first defense capable of filtering attackers' model
updates in a non-IID FL environment. The originality of FedCLEAN is twofold.
First, it relies on a client confidence score derived from the reconstruction
errors of each client's model activation maps for a given trigger set, with
reconstruction errors obtained by means of a Conditional Variational
Autoencoder trained according to a novel server-side strategy. Second, we
propose an ad-hoc trust propagation algorithm based on client scores, which
allows building a cluster of benign clients while flagging potential attackers.
Experimental results on the datasets MNIST and FashionMNIST demonstrate the
robustness of FedCLEAN against Byzantine attackers in non-IID scenarios and a
close-to-zero benign client misclassification rate, even in the absence of an
attack.


http://arxiv.org/abs/2501.12269
Benchmarking Image Perturbations for Testing Automated Driving Assistance Systems. (5%)
Stefano Carlo Lambertenghi; Hannes Leonhard; Andrea Stocco
  Advanced Driver Assistance Systems (ADAS) based on deep neural networks
(DNNs) are widely used in autonomous vehicles for critical perception tasks
such as object detection, semantic segmentation, and lane recognition. However,
these systems are highly sensitive to input variations, such as noise and
changes in lighting, which can compromise their effectiveness and potentially
lead to safety-critical failures.
  This study offers a comprehensive empirical evaluation of image
perturbations, techniques commonly used to assess the robustness of DNNs, to
validate and improve the robustness and generalization of ADAS perception
systems. We first conducted a systematic review of the literature, identifying
38 categories of perturbations. Next, we evaluated their effectiveness in
revealing failures in two different ADAS, both at the component and at the
system level. Finally, we explored the use of perturbation-based data
augmentation and continuous learning strategies to improve ADAS adaptation to
new operational design domains. Our results demonstrate that all categories of
image perturbations successfully expose robustness issues in ADAS and that the
use of dataset augmentation and continuous learning significantly improves ADAS
performance in novel, unseen environments.


http://arxiv.org/abs/2501.12191
A margin-based replacement for cross-entropy loss. (1%)
Michael W. Spratling; Heiko H. Schütt
  Cross-entropy (CE) loss is the de-facto standard for training deep neural
networks to perform classification. However, CE-trained deep neural networks
struggle with robustness and generalisation issues. To alleviate these issues,
we propose high error margin (HEM) loss, a variant of multi-class margin loss
that overcomes the training issues of other margin-based losses. We evaluate
HEM extensively on a range of architectures and datasets. We find that HEM loss
is more effective than cross-entropy loss across a wide range of tasks: unknown
class rejection, adversarial robustness, learning with imbalanced data,
continual learning, and semantic segmentation (a pixel-level classification
task). Despite all training hyper-parameters being chosen for CE loss, HEM is
inferior to CE only in terms of clean accuracy and this difference is
insignificant. We also compare HEM to specialised losses that have previously
been proposed to improve performance on specific tasks. LogitNorm, a loss
achieving state-of-the-art performance on unknown class rejection, produces
similar performance to HEM for this task, but is much poorer for continual
learning and semantic segmentation. Logit-adjusted loss, designed for
imbalanced data, has superior results to HEM for that task, but performs more
poorly on unknown class rejection and semantic segmentation. DICE, a popular
loss for semantic segmentation, is inferior to HEM loss on all tasks, including
semantic segmentation. Thus, HEM often out-performs specialised losses, and in
contrast to them, is a general-purpose replacement for CE loss.


http://arxiv.org/abs/2501.12210
You Can't Eat Your Cake and Have It Too: The Performance Degradation of LLMs with Jailbreak Defense. (1%)
Wuyuao Mai; Geng Hong; Pei Chen; Xudong Pan; Baojun Liu; Yuan Zhang; Haixin Duan; Min Yang
  With the rise of generative large language models (LLMs) like LLaMA and
ChatGPT, these models have significantly transformed daily life and work by
providing advanced insights. However, as jailbreak attacks continue to
circumvent built-in safety mechanisms, exploiting carefully crafted scenarios
or tokens, the safety risks of LLMs have come into focus. While numerous
defense strategies--such as prompt detection, modification, and model
fine-tuning--have been proposed to counter these attacks, a critical question
arises: do these defenses compromise the utility and usability of LLMs for
legitimate users? Existing research predominantly focuses on the effectiveness
of defense strategies without thoroughly examining their impact on performance,
leaving a gap in understanding the trade-offs between LLM safety and
performance. Our research addresses this gap by conducting a comprehensive
study on the utility degradation, safety elevation, and exaggerated-safety
escalation of LLMs with jailbreak defense strategies. We propose USEBench, a
novel benchmark designed to evaluate these aspects, along with USEIndex, a
comprehensive metric for assessing overall model performance. Through
experiments on seven state-of-the-art LLMs, we found that mainstream jailbreak
defenses fail to ensure both safety and performance simultaneously. Although
model-finetuning performs the best overall, their effectiveness varies across
LLMs. Furthermore, vertical comparisons reveal that developers commonly
prioritize performance over safety when iterating or fine-tuning their LLMs.


http://arxiv.org/abs/2501.11568
Graph Defense Diffusion Model. (97%)
Xin He; Wenqi Fan; Yili Wang; Chengyi Liu; Rui Miao; Xin Juan; Xin Wang
  Graph Neural Networks (GNNs) demonstrate significant potential in various
applications but remain highly vulnerable to adversarial attacks, which can
greatly degrade their performance. Existing graph purification methods attempt
to address this issue by filtering attacked graphs; however, they struggle to
effectively defend against multiple types of adversarial attacks simultaneously
due to their limited flexibility, and they lack comprehensive modeling of graph
data due to their heavy reliance on heuristic prior knowledge. To overcome
these challenges, we propose a more versatile approach for defending against
adversarial attacks on graphs. In this work, we introduce the Graph Defense
Diffusion Model (GDDM), a flexible purification method that leverages the
denoising and modeling capabilities of diffusion models. The iterative nature
of diffusion models aligns well with the stepwise process of adversarial
attacks, making them particularly suitable for defense. By iteratively adding
and removing noise, GDDM effectively purifies attacked graphs, restoring their
original structure and features. Our GDDM consists of two key components: (1)
Graph Structure-Driven Refiner, which preserves the basic fidelity of the graph
during the denoising process, and ensures that the generated graph remains
consistent with the original scope; and (2) Node Feature-Constrained
Regularizer, which removes residual impurities from the denoised graph, further
enhances the purification effect. Additionally, we design tailored denoising
strategies to handle different types of adversarial attacks, improving the
model's adaptability to various attack scenarios. Extensive experiments
conducted on three real-world datasets demonstrate that GDDM outperforms
state-of-the-art methods in defending against a wide range of adversarial
attacks, showcasing its robustness and effectiveness.


http://arxiv.org/abs/2501.11848
FedMUA: Exploring the Vulnerabilities of Federated Learning to Malicious Unlearning Attacks. (67%)
Jian Chen; Zehui Lin; Wanyu Lin; Wenlong Shi; Xiaoyan Yin; Di Wang
  Recently, the practical needs of ``the right to be forgotten'' in federated
learning gave birth to a paradigm known as federated unlearning, which enables
the server to forget personal data upon the client's removal request. Existing
studies on federated unlearning have primarily focused on efficiently
eliminating the influence of requested data from the client's model without
retraining from scratch, however, they have rarely doubted the reliability of
the global model posed by the discrepancy between its prediction performance
before and after unlearning. To bridge this gap, we take the first step by
introducing a novel malicious unlearning attack dubbed FedMUA, aiming to unveil
potential vulnerabilities emerging from federated learning during the
unlearning process. The crux of FedMUA is to mislead the global model into
unlearning more information associated with the influential samples for the
target sample than anticipated, thus inducing adverse effects on target samples
from other clients. To achieve this, we design a novel two-step method, known
as Influential Sample Identification and Malicious Unlearning Generation, to
identify and subsequently generate malicious feature unlearning requests within
the influential samples. By doing so, we can significantly alter the
predictions pertaining to the target sample by initiating the malicious feature
unlearning requests, leading to the deliberate manipulation for the user
adversely. Additionally, we design a new defense mechanism that is highly
resilient against malicious unlearning attacks. Extensive experiments on three
realistic datasets reveal that FedMUA effectively induces misclassification on
target samples and can achieve an 80% attack success rate by triggering only
0.3% malicious unlearning requests.


http://arxiv.org/abs/2501.11759
Poison-RAG: Adversarial Data Poisoning Attacks on Retrieval-Augmented Generation in Recommender Systems. (50%)
Fatemeh Nazary; Yashar Deldjoo; Noia Tommaso di
  This study presents Poison-RAG, a framework for adversarial data poisoning
attacks targeting retrieval-augmented generation (RAG)-based recommender
systems. Poison-RAG manipulates item metadata, such as tags and descriptions,
to influence recommendation outcomes. Using item metadata generated through a
large language model (LLM) and embeddings derived via the OpenAI API, we
explore the impact of adversarial poisoning attacks on provider-side, where
attacks are designed to promote long-tail items and demote popular ones. Two
attack strategies are proposed: local modifications, which personalize tags for
each item using BERT embeddings, and global modifications, applying uniform
tags across the dataset. Experiments conducted on the MovieLens dataset in a
black-box setting reveal that local strategies improve manipulation
effectiveness by up to 50\%, while global strategies risk boosting already
popular items. Results indicate that popular items are more susceptible to
attacks, whereas long-tail items are harder to manipulate. Approximately 70\%
of items lack tags, presenting a cold-start challenge; data augmentation and
synthesis are proposed as potential defense mechanisms to enhance RAG-based
systems' resilience. The findings emphasize the need for robust metadata
management to safeguard recommendation frameworks. Code and data are available
at https://github.com/atenanaz/Poison-RAG.


http://arxiv.org/abs/2501.11462
On the Adversarial Vulnerabilities of Transfer Learning in Remote Sensing. (41%)
Tao Bai; Xingjian Tian; Yonghao Xu; Bihan Wen
  The use of pretrained models from general computer vision tasks is widespread
in remote sensing, significantly reducing training costs and improving
performance. However, this practice also introduces vulnerabilities to
downstream tasks, where publicly available pretrained models can be used as a
proxy to compromise downstream models. This paper presents a novel Adversarial
Neuron Manipulation method, which generates transferable perturbations by
selectively manipulating single or multiple neurons in pretrained models.
Unlike existing attacks, this method eliminates the need for domain-specific
information, making it more broadly applicable and efficient. By targeting
multiple fragile neurons, the perturbations achieve superior attack
performance, revealing critical vulnerabilities in deep learning models.
Experiments on diverse models and remote sensing datasets validate the
effectiveness of the proposed method. This low-access adversarial neuron
manipulation technique highlights a significant security risk in transfer
learning models, emphasizing the urgent need for more robust defenses in their
design when addressing the safety-critical remote sensing tasks.


http://arxiv.org/abs/2501.11852
Cross-Entropy Attacks to Language Models via Rare Event Simulation. (12%)
Mingze Ni; Yongshun Gong; Wei Liu
  Black-box textual adversarial attacks are challenging due to the lack of
model information and the discrete, non-differentiable nature of text. Existing
methods often lack versatility for attacking different models, suffer from
limited attacking performance due to the inefficient optimization with word
saliency ranking, and frequently sacrifice semantic integrity to achieve better
attack outcomes. This paper introduces a novel approach to textual adversarial
attacks, which we call Cross-Entropy Attacks (CEA), that uses Cross-Entropy
optimization to address the above issues. Our CEA approach defines adversarial
objectives for both soft-label and hard-label settings and employs CE
optimization to identify optimal replacements. Through extensive experiments on
document classification and language translation problems, we demonstrate that
our attack method excels in terms of attacking performance, imperceptibility,
and sentence quality.


http://arxiv.org/abs/2501.11577
Rethinking Membership Inference Attacks Against Transfer Learning. (9%)
Cong Wu; Jing Chen; Qianru Fang; Kun He; Ziming Zhao; Hao Ren; Guowen Xu; Yang Liu; Yang Xiang
  Transfer learning, successful in knowledge translation across related tasks,
faces a substantial privacy threat from membership inference attacks (MIAs).
These attacks, despite posing significant risk to ML model's training data,
remain limited-explored in transfer learning. The interaction between teacher
and student models in transfer learning has not been thoroughly explored in
MIAs, potentially resulting in an under-examined aspect of privacy
vulnerabilities within transfer learning. In this paper, we propose a new MIA
vector against transfer learning, to determine whether a specific data point
was used to train the teacher model while only accessing the student model in a
white-box setting. Our method delves into the intricate relationship between
teacher and student models, analyzing the discrepancies in hidden layer
representations between the student model and its shadow counterpart. These
identified differences are then adeptly utilized to refine the shadow model's
training process and to inform membership inference decisions effectively. Our
method, evaluated across four datasets in diverse transfer learning tasks,
reveals that even when an attacker only has access to the student model, the
teacher model's training data remains susceptible to MIAs. We believe our work
unveils the unexplored risk of membership inference in transfer learning.


http://arxiv.org/abs/2501.11815
CogMorph: Cognitive Morphing Attacks for Text-to-Image Models. (1%)
Zonglei Jing; Zonghao Ying; Le Wang; Siyuan Liang; Aishan Liu; Xianglong Liu; Dacheng Tao
  The development of text-to-image (T2I) generative models, that enable the
creation of high-quality synthetic images from textual prompts, has opened new
frontiers in creative design and content generation. However, this paper
reveals a significant and previously unrecognized ethical risk inherent in this
technology and introduces a novel method, termed the Cognitive Morphing Attack
(CogMorph), which manipulates T2I models to generate images that retain the
original core subjects but embeds toxic or harmful contextual elements. This
nuanced manipulation exploits the cognitive principle that human perception of
concepts is shaped by the entire visual scene and its context, producing images
that amplify emotional harm far beyond attacks that merely preserve the
original semantics. To address this, we first construct an imagery toxicity
taxonomy spanning 10 major and 48 sub-categories, aligned with human
cognitive-perceptual dimensions, and further build a toxicity risk matrix
resulting in 1,176 high-quality T2I toxic prompts. Based on this, our CogMorph
first introduces Cognitive Toxicity Augmentation, which develops a cognitive
toxicity knowledge base with rich external toxic representations for humans
(e.g., fine-grained visual features) that can be utilized to further guide the
optimization of adversarial prompts. In addition, we present Contextual
Hierarchical Morphing, which hierarchically extracts critical parts of the
original prompt (e.g., scenes, subjects, and body parts), and then iteratively
retrieves and fuses toxic features to inject harmful contexts. Extensive
experiments on multiple open-sourced T2I models and black-box commercial APIs
(e.g., DALLE-3) demonstrate the efficacy of CogMorph which significantly
outperforms other baselines by large margins (+20.62% on average).


http://arxiv.org/abs/2501.10996
Effectiveness of Adversarial Benign and Malware Examples in Evasion and Poisoning Attacks. (99%)
Matouš Kozák; Martin Jureček
  Adversarial attacks present significant challenges for malware detection
systems. This research investigates the effectiveness of benign and malicious
adversarial examples (AEs) in evasion and poisoning attacks on the Portable
Executable file domain. A novel focus of this study is on benign AEs, which,
although not directly harmful, can increase false positives and undermine trust
in antivirus solutions. We propose modifying existing adversarial malware
generators to produce benign AEs and show they are as successful as malware AEs
in evasion attacks. Furthermore, our data show that benign AEs have a more
decisive influence in poisoning attacks than standard malware AEs,
demonstrating their superior ability to decrease the model's performance. Our
findings introduce new opportunities for adversaries and further increase the
attack surface that needs to be protected by security researchers.


http://arxiv.org/abs/2501.10985
GRID: Protecting Training Graph from Link Stealing Attacks on GNN Models. (12%)
Jiadong Lou; Xu Yuan; Rui Zhang; Xingliang Yuan; Neil Gong; Nian-Feng Tzeng
  Graph neural networks (GNNs) have exhibited superior performance in various
classification tasks on graph-structured data. However, they encounter the
potential vulnerability from the link stealing attacks, which can infer the
presence of a link between two nodes via measuring the similarity of its
incident nodes' prediction vectors produced by a GNN model. Such attacks pose
severe security and privacy threats to the training graph used in GNN models.
In this work, we propose a novel solution, called Graph Link Disguise (GRID),
to defend against link stealing attacks with the formal guarantee of GNN model
utility for retaining prediction accuracy. The key idea of GRID is to add
carefully crafted noises to the nodes' prediction vectors for disguising
adjacent nodes as n-hop indirect neighboring nodes. We take into account the
graph topology and select only a subset of nodes (called core nodes) covering
all links for adding noises, which can avert the noises offset and have the
further advantages of reducing both the distortion loss and the computation
cost. Our crafted noises can ensure 1) the noisy prediction vectors of any two
adjacent nodes have their similarity level like that of two non-adjacent nodes
and 2) the model prediction is unchanged to ensure zero utility loss. Extensive
experiments on five datasets are conducted to show the effectiveness of our
proposed GRID solution against different representative link-stealing attacks
under transductive settings and inductive settings respectively, as well as two
influence-based attacks. Meanwhile, it achieves a much better privacy-utility
trade-off than existing methods when extended to GNNs.


http://arxiv.org/abs/2501.11183
Can Safety Fine-Tuning Be More Principled? Lessons Learned from Cybersecurity. (10%)
David Williams-King; Linh Le; Adam Oberman; Yoshua Bengio
  As LLMs develop increasingly advanced capabilities, there is an increased
need to minimize the harm that could be caused to society by certain model
outputs; hence, most LLMs have safety guardrails added, for example via
fine-tuning. In this paper, we argue the position that current safety
fine-tuning is very similar to a traditional cat-and-mouse game (or arms race)
between attackers and defenders in cybersecurity. Model jailbreaks and attacks
are patched with bandaids to target the specific attack mechanism, but many
similar attack vectors might remain. When defenders are not proactively coming
up with principled mechanisms, it becomes very easy for attackers to sidestep
any new defenses. We show how current defenses are insufficient to prevent new
adversarial jailbreak attacks, reward hacking, and loss of control problems. In
order to learn from past mistakes in cybersecurity, we draw analogies with
historical examples and develop lessons learned that can be applied to LLM
safety. These arguments support the need for new and more principled approaches
to designing safe models, which are architected for security from the
beginning. We describe several such approaches from the AI literature.


http://arxiv.org/abs/2501.13115
Dagger Behind Smile: Fool LLMs with a Happy Ending Story. (4%)
Xurui Song; Zhixin Xie; Shuo Huai; Jiayi Kong; Jun Luo
  The wide adoption of Large Language Models (LLMs) has attracted significant
attention from \textit{jailbreak} attacks, where adversarial prompts crafted
through optimization or manual design exploit LLMs to generate malicious
content. However, optimization-based attacks have limited efficiency and
transferability, while manual designs are either easily detectable or demand
intricate interactions with LLMs. In this paper, we first point out a novel
perspective for jailbreak attacks: LLMs are more responsive to
\textit{positive} prompts. Based on this, we deploy Happy Ending Attack (HEA)
to wrap up a malicious request in a scenario template involving a positive
prompt formed mainly via a \textit{happy ending}, it thus fools LLMs into
jailbreaking either immediately or at a follow-up malicious request. This has
made HEA both efficient and effective, as it requires only up to two steps to
fully jailbreak LLMs. Extensive experiments show that our HEA can successfully
jailbreak on state-of-the-art LLMs, including GPT-4o, Llama3-70b, Gemini-pro,
and achieves 88.79\% Attack Success Rate on average. We also provide potential
quantitative explanations for the success of HEA.


http://arxiv.org/abs/2501.11054
Temporal Analysis of Adversarial Attacks in Federated Learning. (2%)
Rohit Mapakshi; Sayma Akther; Mark Stamp
  In this paper, we experimentally analyze the robustness of selected Federated
Learning (FL) systems in the presence of adversarial clients. We find that
temporal attacks significantly affect model performance in the FL models
tested, especially when the adversaries are active throughout or during the
later rounds. We consider a variety of classic learning models, including
Multinominal Logistic Regression (MLR), Random Forest, XGBoost, Support Vector
Classifier (SVC), as well as various Neural Network models including Multilayer
Perceptron (MLP), Convolution Neural Network (CNN), Recurrent Neural Network
(RNN), and Long Short-Term Memory (LSTM). Our results highlight the
effectiveness of temporal attacks and the need to develop strategies to make
the FL process more robust against such attacks. We also briefly consider the
effectiveness of defense mechanisms, including outlier detection in the
aggregation algorithm.


http://arxiv.org/abs/2501.11171
Counteracting temporal attacks in Video Copy Detection. (1%)
Katarzyna Fojcik; Piotr Syga
  Video Copy Detection (VCD) plays a crucial role in copyright protection and
content verification by identifying duplicates and near-duplicates in
large-scale video databases. The META AI Challenge on video copy detection
provided a benchmark for evaluating state-of-the-art methods, with the
Dual-level detection approach emerging as a winning solution. This method
integrates Video Editing Detection and Frame Scene Detection to handle
adversarial transformations and large datasets efficiently. However, our
analysis reveals significant limitations in the VED component, particularly in
its ability to handle exact copies. Moreover, Dual-level detection shows
vulnerability to temporal attacks. To address it, we propose an improved frame
selection strategy based on local maxima of interframe differences, which
enhances robustness against adversarial temporal modifications while
significantly reducing computational overhead. Our method achieves an increase
of 1.4 to 5.8 times in efficiency over the standard 1 FPS approach. Compared to
Dual-level detection method, our approach maintains comparable micro-average
precision ($\mu$AP) while also demonstrating improved robustness against
temporal attacks. Given 56\% reduced representation size and the inference time
of more than 2 times faster, our approach is more suitable to real-world
resource restriction.


http://arxiv.org/abs/2501.10906
Explainable Adversarial Attacks on Coarse-to-Fine Classifiers. (86%)
Akram Heidarizadeh; Connor Hatfield; Lorenzo Lazzarotto; HanQin Cai; George Atia
  Traditional adversarial attacks typically aim to alter the predicted labels
of input images by generating perturbations that are imperceptible to the human
eye. However, these approaches often lack explainability. Moreover, most
existing work on adversarial attacks focuses on single-stage classifiers, but
multi-stage classifiers are largely unexplored. In this paper, we introduce
instance-based adversarial attacks for multi-stage classifiers, leveraging
Layer-wise Relevance Propagation (LRP), which assigns relevance scores to
pixels based on their influence on classification outcomes. Our approach
generates explainable adversarial perturbations by utilizing LRP to identify
and target key features critical for both coarse and fine-grained
classifications. Unlike conventional attacks, our method not only induces
misclassification but also enhances the interpretability of the model's
behavior across classification stages, as demonstrated by experimental results.


http://arxiv.org/abs/2501.10740
Stability of neural ODEs by a control over the expansivity of their flows. (26%)
Marinis Arturo De; Nicola Guglielmi; Stefano Sicilia; Francesco Tudisco
  We propose a method to enhance the stability of a neural ordinary
differential equation (neural ODE) by means of a control over the Lipschitz
constant $C$ of its flow. Since it is known that $C$ depends on the logarithmic
norm of the Jacobian matrix associated with the neural ODE, we tune this
parameter at our convenience by suitably perturbing the Jacobian matrix with a
perturbation as small as possible in Frobenius norm. We do so by introducing an
optimization problem for which we propose a nested two-level algorithm. For a
given perturbation size, the inner level computes the optimal perturbation with
a fixed Frobenius norm, while the outer level tunes the perturbation amplitude.
We embed the proposed algorithm in the training of the neural ODE to improve
its stability. Numerical experiments on the MNIST and FashionMNIST datasets
show that an image classifier including a neural ODE in its architecture
trained according to our strategy is more stable than the same classifier
trained in the classical way, and therefore, it is more robust and less
vulnerable to adversarial attacks.


http://arxiv.org/abs/2501.10817
A comprehensive survey on RPL routing-based attacks, defences and future directions in Internet of Things. (12%)
Anil K Prajapati; Emmanuel S Pilli; Ramesh B Battula; Vijay Varadharajan; Abhishek Verma; R C Joshi
  The Internet of Things (IoT) is a network of digital devices like sensors,
processors, embedded and communication devices that can connect to and exchange
data with other devices and systems over the internet. IoT devices have
limitations on power, memory, and computational resources. Researchers have
developed the IPv6 Over Low-power Wireless Personal Area Network (6LoWPAN)
protocols to provide wireless connectivity among these devices while overcoming
the constraints on resources. 6LoWPAN has been approved subsequently by the
Internet Engineering Task Force (IETF). The IETF Routing Over Low-power and
Lossy Networks (ROLL) standardized the Routing Protocol for LLNs known as RPL
(IETF RFC 6550), which is part of the 6LoWPAN stack. However, IoT devices are
vulnerable to various attacks on RPL-based routing. This survey provides an in
depth study of existing RPL-based attacks and defense published from year 2011
to 2024 from highly reputed journals and conferences. By thematic analysis of
existing routing attacks on RPL, we developed a novel attack taxonomy which
focuses on the nature of routing attacks and classifies them into 12 major
categories. Subsequently, the impact of each attack on the network is analyzed
and discussed real life scenarios of these attacks. Another contribution of
this survey proposed a novel taxonomy for classification of defense mechanisms
into 8 major categories against routing attacks based on type of defense
strategy. The detailed analysis of each defense mechanism with real life
applicability is explained. Furthermore, evaluation tools such as testbeds and
simulators for RPL-based attack and defense are discussed and critically
analyzed in terms of real world applicability. Finally, open research
challenges are presented on the basis of research gaps of existing literature
along with research directions for practitioners and researchers.


http://arxiv.org/abs/2501.10013
CaFA: Cost-aware, Feasible Attacks With Database Constraints Against Neural Tabular Classifiers. (99%)
Matan Ben-Tov; Daniel Deutch; Nave Frost; Mahmood Sharif
  This work presents CaFA, a system for Cost-aware Feasible Attacks for
assessing the robustness of neural tabular classifiers against adversarial
examples realizable in the problem space, while minimizing adversaries' effort.
To this end, CaFA leverages TabPGD$-$an algorithm we set forth to generate
adversarial perturbations suitable for tabular data$-$ and incorporates
integrity constraints automatically mined by state-of-the-art database methods.
After producing adversarial examples in the feature space via TabPGD, CaFA
projects them on the mined constraints, leading, in turn, to better attack
realizability. We tested CaFA with three datasets and two architectures and
found, among others, that the constraints we use are of higher quality
(measured via soundness and completeness) than ones employed in prior work.
Moreover, CaFA achieves higher feasible success rates$-$i.e., it generates
adversarial examples that are often misclassified while satisfying
constraints$-$than prior attacks while simultaneously perturbing few features
with lower magnitudes, thus saving effort and improving inconspicuousness. We
open-source CaFA, hoping it will serve as a generic system enabling
machine-learning engineers to assess their models' robustness against
realizable attacks, thus advancing deployed models' trustworthiness.


http://arxiv.org/abs/2501.10606
Differentiable Adversarial Attacks for Marked Temporal Point Processes. (86%)
Pritish Chakraborty; Vinayak Gupta; Rahul R; Srikanta J. Bedathur; Abir De
  Marked temporal point processes (MTPPs) have been shown to be extremely
effective in modeling continuous time event sequences (CTESs). In this work, we
present adversarial attacks designed specifically for MTPP models. A key
criterion for a good adversarial attack is its imperceptibility. For objects
such as images or text, this is often achieved by bounding perturbation in some
fixed $L_p$ norm-ball. However, similarly minimizing distance norms between two
CTESs in the context of MTPPs is challenging due to their sequential nature and
varying time-scales and lengths. We address this challenge by first permuting
the events and then incorporating the additive noise to the arrival timestamps.
However, the worst case optimization of such adversarial attacks is a hard
combinatorial problem, requiring exploration across a permutation space that is
factorially large in the length of the input sequence. As a result, we propose
a novel differentiable scheme PERMTPP using which we can perform adversarial
attacks by learning to minimize the likelihood, while minimizing the distance
between two CTESs. Our experiments on four real-world datasets demonstrate the
offensive and defensive capabilities, and lower inference times of PERMTPP.


http://arxiv.org/abs/2501.10639
Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attacks. (50%)
Xin Yi; Yue Li; Linlin Wang; Xiaoling Wang; Liang He
  Ensuring safety alignment has become a critical requirement for large
language models (LLMs), particularly given their widespread deployment in
real-world applications. However, LLMs remain susceptible to jailbreak attacks,
which exploit system vulnerabilities to bypass safety measures and generate
harmful outputs. Although numerous defense mechanisms based on adversarial
training have been proposed, a persistent challenge lies in the exacerbation of
over-refusal behaviors, which compromise the overall utility of the model. To
address these challenges, we propose a Latent-space Adversarial Training with
Post-aware Calibration (LATPC) framework. During the adversarial training
phase, LATPC compares harmful and harmless instructions in the latent space and
extracts safety-critical dimensions to construct refusal features attack,
precisely simulating agnostic jailbreak attack types requiring adversarial
mitigation. At the inference stage, an embedding-level calibration mechanism is
employed to alleviate over-refusal behaviors with minimal computational
overhead. Experimental results demonstrate that, compared to various defense
methods across five types of jailbreak attacks, LATPC framework achieves a
superior balance between safety and utility. Moreover, our analysis underscores
the effectiveness of extracting safety-critical dimensions from the latent
space for constructing robust refusal feature attacks.


http://arxiv.org/abs/2501.09446
Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness. (98%)
Zeyu Wang; Cihang Xie; Brian Bartoldson; Bhavya Kailkhura
  This paper investigates the robustness of vision-language models against
adversarial visual perturbations and introduces a novel ``double visual
defense" to enhance this robustness. Unlike previous approaches that resort to
lightweight adversarial fine-tuning of a pre-trained CLIP model, we perform
large-scale adversarial vision-language pre-training from scratch using
web-scale data. We then strengthen the defense by incorporating adversarial
visual instruction tuning. The resulting models from each stage, $\Delta$CLIP
and $\Delta^2$LLaVA, show substantially enhanced zero-shot robustness and set a
new state-of-the-art in adversarial defense for vision-language models. For
example, the adversarial robustness of $\Delta$CLIP surpasses that of the
previous best models on ImageNet-1k by ~20%. %For example, $\Delta$CLIP
surpasses the previous best models on ImageNet-1k by ~20% in terms of
adversarial robustness. Similarly, compared to prior art, $\Delta^2$LLaVA
brings a ~30% robustness improvement to image captioning task and a ~20%
robustness improvement to visual question answering task. Furthermore, our
models exhibit stronger zero-shot recognition capability, fewer hallucinations,
and superior reasoning performance compared to baselines. Our project page is
https://doublevisualdefense.github.io/.


http://arxiv.org/abs/2501.09609
Adversarial-Ensemble Kolmogorov Arnold Networks for Enhancing Indoor Wi-Fi Positioning: A Defensive Approach Against Spoofing and Signal Manipulation Attacks. (80%)
Mitul Goswami; Romit Chatterjee; Somnath Mahato; Prasant Kumar Pattnaik
  The research presents a study on enhancing the robustness of Wi-Fi-based
indoor positioning systems against adversarial attacks. The goal is to improve
the positioning accuracy and resilience of these systems under two attack
scenarios: Wi-Fi Spoofing and Signal Strength Manipulation. Three models are
developed and evaluated: a baseline model (M_Base), an adversarially trained
robust model (M_Rob), and an ensemble model (M_Ens). All models utilize a
Kolmogorov-Arnold Network (KAN) architecture. The robust model is trained with
adversarially perturbed data, while the ensemble model combines predictions
from both the base and robust models. Experimental results show that the robust
model reduces positioning error by approximately 10% compared to the baseline,
achieving 2.03 meters error under Wi-Fi spoofing and 2.00 meters under signal
strength manipulation. The ensemble model further outperforms with errors of
2.01 meters and 1.975 meters for the respective attack types. This analysis
highlights the effectiveness of adversarial training techniques in mitigating
attack impacts. The findings underscore the importance of considering
adversarial scenarios in developing indoor positioning systems, as improved
resilience can significantly enhance the accuracy and reliability of such
systems in mission-critical environments.


http://arxiv.org/abs/2501.09320
Cooperative Decentralized Backdoor Attacks on Vertical Federated Learning. (31%)
Seohyun Lee; Wenzhi Fang; Anindya Bijoy Das; Seyyedali Hosseinalipour; David J. Love; Christopher G. Brinton
  Federated learning (FL) is vulnerable to backdoor attacks, where adversaries
alter model behavior on target classification labels by embedding triggers into
data samples. While these attacks have received considerable attention in
horizontal FL, they are less understood for vertical FL (VFL), where devices
hold different features of the samples, and only the server holds the labels.
In this work, we propose a novel backdoor attack on VFL which (i) does not rely
on gradient information from the server and (ii) considers potential collusion
among multiple adversaries for sample selection and trigger embedding. Our
label inference model augments variational autoencoders with metric learning,
which adversaries can train locally. A consensus process over the adversary
graph topology determines which datapoints to poison. We further propose
methods for trigger splitting across the adversaries, with an intensity-based
implantation scheme skewing the server towards the trigger. Our convergence
analysis reveals the impact of backdoor perturbations on VFL indicated by a
stationarity gap for the trained model, which we verify empirically as well. We
conduct experiments comparing our attack with recent backdoor VFL approaches,
finding that ours obtains significantly higher success rates for the same main
task performance despite not using server information. Additionally, our
results verify the impact of collusion on attack performance.


http://arxiv.org/abs/2501.09798
Computing Optimization-Based Prompt Injections Against Closed-Weights Models By Misusing a Fine-Tuning API. (2%)
Andrey Labunets; Nishit V. Pandya; Ashish Hooda; Xiaohan Fu; Earlence Fernandes
  We surface a new threat to closed-weight Large Language Models (LLMs) that
enables an attacker to compute optimization-based prompt injections.
Specifically, we characterize how an attacker can leverage the loss-like
information returned from the remote fine-tuning interface to guide the search
for adversarial prompts. The fine-tuning interface is hosted by an LLM vendor
and allows developers to fine-tune LLMs for their tasks, thus providing
utility, but also exposes enough information for an attacker to compute
adversarial prompts. Through an experimental analysis, we characterize the
loss-like values returned by the Gemini fine-tuning API and demonstrate that
they provide a useful signal for discrete optimization of adversarial prompts
using a greedy search algorithm. Using the PurpleLlama prompt injection
benchmark, we demonstrate attack success rates between 65% and 82% on Google's
Gemini family of LLMs. These attacks exploit the classic utility-security
tradeoff - the fine-tuning interface provides a useful feature for developers
but also exposes the LLMs to powerful attacks.


http://arxiv.org/abs/2501.09328
Neural Honeytrace: A Robust Plug-and-Play Watermarking Framework against Model Extraction Attacks. (1%)
Yixiao Xu; Binxing Fang; Rui Wang; Yinghai Zhou; Shouling Ji; Yuan Liu; Mohan Li; Zhihong Tian
  Developing high-performance deep learning models is resource-intensive,
leading model owners to utilize Machine Learning as a Service (MLaaS) platforms
instead of publicly releasing their models. However, malicious users may
exploit query interfaces to execute model extraction attacks, reconstructing
the target model's functionality locally. While prior research has investigated
triggerable watermarking techniques for asserting ownership, existing methods
face significant challenges: (1) most approaches require additional training,
resulting in high overhead and limited flexibility, and (2) they often fail to
account for advanced attackers, leaving them vulnerable to adaptive attacks.
  In this paper, we propose Neural Honeytrace, a robust plug-and-play
watermarking framework against model extraction attacks. We first formulate a
watermark transmission model from an information-theoretic perspective,
providing an interpretable account of the principles and limitations of
existing triggerable watermarking. Guided by the model, we further introduce:
(1) a similarity-based training-free watermarking method for plug-and-play and
flexible watermarking, and (2) a distribution-based multi-step watermark
information transmission strategy for robust watermarking. Comprehensive
experiments on four datasets demonstrate that Neural Honeytrace outperforms
previous methods in efficiency and resisting adaptive attacks. Neural
Honeytrace reduces the average number of samples required for a worst-case
t-Test-based copyright claim from $12,000$ to $200$ with zero training cost.


http://arxiv.org/abs/2501.09086
Salient Information Preserving Adversarial Training Improves Clean and Robust Accuracy. (98%)
Timothy Redgrave; Adam Czajka
  In this work we introduce Salient Information Preserving Adversarial Training
(SIP-AT), an intuitive method for relieving the robustness-accuracy trade-off
incurred by traditional adversarial training. SIP-AT uses salient image regions
to guide the adversarial training process in such a way that fragile features
deemed meaningful by an annotator remain unperturbed during training, allowing
models to learn highly predictive non-robust features without sacrificing
overall robustness. This technique is compatible with both human-based and
automatically generated salience estimates, allowing SIP-AT to be used as a
part of human-driven model development without forcing SIP-AT to be reliant
upon additional human data. We perform experiments across multiple datasets and
architectures and demonstrate that SIP-AT is able to boost the clean accuracy
of models while maintaining a high degree of robustness against attacks at
multiple epsilon levels. We complement our central experiments with an
observational study measuring the rate at which human subjects successfully
identify perturbed images. This study helps build a more intuitive
understanding of adversarial attack strength and demonstrates the heightened
importance of low-epsilon robustness. Our results demonstrate the efficacy of
SIP-AT and provide valuable insight into the risks posed by adversarial samples
of various strengths.


http://arxiv.org/abs/2501.08862
ARMOR: Shielding Unlearnable Examples against Data Augmentation. (75%)
Xueluan Gong; Yuji Wang; Yanjiao Chen; Haocheng Dong; Yiming Li; Mengyuan Sun; Shuaike Li; Qian Wang; Chen Chen
  Private data, when published online, may be collected by unauthorized parties
to train deep neural networks (DNNs). To protect privacy, defensive noises can
be added to original samples to degrade their learnability by DNNs. Recently,
unlearnable examples are proposed to minimize the training loss such that the
model learns almost nothing. However, raw data are often pre-processed before
being used for training, which may restore the private information of protected
data. In this paper, we reveal the data privacy violation induced by data
augmentation, a commonly used data pre-processing technique to improve model
generalization capability, which is the first of its kind as far as we are
concerned. We demonstrate that data augmentation can significantly raise the
accuracy of the model trained on unlearnable examples from 21.3% to 66.1%. To
address this issue, we propose a defense framework, dubbed ARMOR, to protect
data privacy from potential breaches of data augmentation. To overcome the
difficulty of having no access to the model training process, we design a
non-local module-assisted surrogate model that better captures the effect of
data augmentation. In addition, we design a surrogate augmentation selection
strategy that maximizes distribution alignment between augmented and
non-augmented samples, to choose the optimal augmentation strategy for each
class. We also use a dynamic step size adjustment algorithm to enhance the
defensive noise generation process. Extensive experiments are conducted on 4
datasets and 5 data augmentation methods to verify the performance of ARMOR.
Comparisons with 6 state-of-the-art defense methods have demonstrated that
ARMOR can preserve the unlearnability of protected private data under data
augmentation. ARMOR reduces the test accuracy of the model trained on augmented
protected samples by as much as 60% more than baselines.


http://arxiv.org/abs/2501.09006
Improving Stability Estimates in Adversarial Explainable AI through Alternate Search Methods. (13%)
Christopher Burger; Charles Walter
  Advances in the effectiveness of machine learning models have come at the
cost of enormous complexity resulting in a poor understanding of how they
function. Local surrogate methods have been used to approximate the workings of
these complex models, but recent work has revealed their vulnerability to
adversarial attacks where the explanation produced is appreciably different
while the meaning and structure of the complex model's output remains similar.
This prior work has focused on the existence of these weaknesses but not on
their magnitude. Here we explore using an alternate search method with the goal
of finding minimum viable perturbations, the fewest perturbations necessary to
achieve a fixed similarity value between the original and altered text's
explanation. Intuitively, a method that requires fewer perturbations to expose
a given level of instability is inferior to one which requires more. This
nuance allows for superior comparisons of the stability of explainability
methods.


http://arxiv.org/abs/2501.10466
Improving the Efficiency of Self-Supervised Adversarial Training through Latent Clustering-Based Selection. (12%)
Somrita Ghosh; Yuelin Xu; Xiao Zhang
  Compared with standard learning, adversarially robust learning is widely
recognized to demand significantly more training examples. Recent works propose
the use of self-supervised adversarial training (SSAT) with external or
synthetically generated unlabeled data to enhance model robustness. However,
SSAT requires a substantial amount of extra unlabeled data, significantly
increasing memory usage and model training times. To address these challenges,
we propose novel methods to strategically select a small subset of unlabeled
data essential for SSAT and robustness improvement. Our selection prioritizes
data points near the model's decision boundary based on latent clustering-based
techniques, efficiently identifying a critical subset of unlabeled data with a
higher concentration of boundary-adjacent points. While focusing on
near-boundary data, our methods are designed to maintain a balanced ratio
between boundary and non-boundary data points to avoid overfitting. Our
experiments on image benchmarks show that integrating our selection strategies
into self-supervised adversarial training can largely reduce memory and
computational requirements while achieving high model robustness. In
particular, our latent clustering-based selection method with k-means is the
most effective, achieving nearly identical test-time robust accuracies with 5
to 10 times less external or generated unlabeled data when applied to image
benchmarks. Additionally, we validate the generalizability of our approach
across various application scenarios, including a real-world medical dataset
for COVID-19 chest X-ray classification.


http://arxiv.org/abs/2501.07922
VENOM: Text-driven Unrestricted Adversarial Example Generation with Diffusion Models. (99%)
Hui Kuurila-Zhang; Haoyu Chen; Guoying Zhao
  Adversarial attacks have proven effective in deceiving machine learning
models by subtly altering input images, motivating extensive research in recent
years. Traditional methods constrain perturbations within $l_p$-norm bounds,
but advancements in Unrestricted Adversarial Examples (UAEs) allow for more
complex, generative-model-based manipulations. Diffusion models now lead UAE
generation due to superior stability and image quality over GANs. However,
existing diffusion-based UAE methods are limited to using reference images and
face challenges in generating Natural Adversarial Examples (NAEs) directly from
random noise, often producing uncontrolled or distorted outputs. In this work,
we introduce VENOM, the first text-driven framework for high-quality
unrestricted adversarial examples generation through diffusion models. VENOM
unifies image content generation and adversarial synthesis into a single
reverse diffusion process, enabling high-fidelity adversarial examples without
sacrificing attack success rate (ASR). To stabilize this process, we
incorporate an adaptive adversarial guidance strategy with momentum, ensuring
that the generated adversarial examples $x^*$ align with the distribution
$p(x)$ of natural images. Extensive experiments demonstrate that VENOM achieves
superior ASR and image quality compared to prior methods, marking a significant
advancement in adversarial example generation and providing insights into model
vulnerabilities for improved defense development.


http://arxiv.org/abs/2501.08415
Cross-Modal Transferable Image-to-Video Attack on Video Quality Metrics. (99%)
Georgii Gotin; Ekaterina Shumitskaya; Anastasia Antsiferova; Dmitriy Vatolin
  Recent studies have revealed that modern image and video quality assessment
(IQA/VQA) metrics are vulnerable to adversarial attacks. An attacker can
manipulate a video through preprocessing to artificially increase its quality
score according to a certain metric, despite no actual improvement in visual
quality. Most of the attacks studied in the literature are white-box attacks,
while black-box attacks in the context of VQA have received less attention.
Moreover, some research indicates a lack of transferability of adversarial
examples generated for one model to another when applied to VQA. In this paper,
we propose a cross-modal attack method, IC2VQA, aimed at exploring the
vulnerabilities of modern VQA models. This approach is motivated by the
observation that the low-level feature spaces of images and videos are similar.
We investigate the transferability of adversarial perturbations across
different modalities; specifically, we analyze how adversarial perturbations
generated on a white-box IQA model with an additional CLIP module can
effectively target a VQA model. The addition of the CLIP module serves as a
valuable aid in increasing transferability, as the CLIP model is known for its
effective capture of low-level semantics. Extensive experiments demonstrate
that IC2VQA achieves a high success rate in attacking three black-box VQA
models. We compare our method with existing black-box attack strategies,
highlighting its superiority in terms of attack success within the same number
of iterations and levels of attack strength. We believe that the proposed
method will contribute to the deeper analysis of robust VQA metrics.


http://arxiv.org/abs/2501.08258
Towards an End-to-End (E2E) Adversarial Learning and Application in the Physical World. (97%)
Dudi Biton; Jacob Shams; Satoru Koda; Asaf Shabtai; Yuval Elovici; Ben Nassi
  The traditional learning process of patch-based adversarial attacks,
conducted in the digital domain and then applied in the physical domain (e.g.,
via printed stickers), may suffer from reduced performance due to adversarial
patches' limited transferability from the digital domain to the physical
domain. Given that previous studies have considered using projectors to apply
adversarial attacks, we raise the following question: can adversarial learning
(i.e., patch generation) be performed entirely in the physical domain with a
projector? In this work, we propose the Physical-domain Adversarial Patch
Learning Augmentation (PAPLA) framework, a novel end-to-end (E2E) framework
that converts adversarial learning from the digital domain to the physical
domain using a projector. We evaluate PAPLA across multiple scenarios,
including controlled laboratory settings and realistic outdoor environments,
demonstrating its ability to ensure attack success compared to conventional
digital learning-physical application (DL-PA) methods. We also analyze the
impact of environmental factors, such as projection surface color, projector
strength, ambient light, distance, and angle of the target object relative to
the camera, on the effectiveness of projected patches. Finally, we demonstrate
the feasibility of the attack against a parked car and a stop sign in a
real-world outdoor environment. Our results show that under specific
conditions, E2E adversarial learning in the physical domain eliminates the
transferability issue and ensures evasion by object detectors. Finally, we
provide insights into the challenges and opportunities of applying adversarial
learning in the physical domain and explain where such an approach is more
effective than using a sticker.


http://arxiv.org/abs/2501.08152
Energy Backdoor Attack to Deep Neural Networks. (64%)
Hanene F. Z. Brachemi Meftah; Wassim Hamidouche; Sid Ahmed Fezza; Olivier Déforges; Kassem Kallas
  The rise of deep learning (DL) has increased computing complexity and energy
use, prompting the adoption of application specific integrated circuits (ASICs)
for energy-efficient edge and mobile deployment. However, recent studies have
demonstrated the vulnerability of these accelerators to energy attacks. Despite
the development of various inference time energy attacks in prior research,
backdoor energy attacks remain unexplored. In this paper, we design an
innovative energy backdoor attack against deep neural networks (DNNs) operating
on sparsity-based accelerators. Our attack is carried out in two distinct
phases: backdoor injection and backdoor stealthiness. Experimental results
using ResNet-18 and MobileNet-V2 models trained on CIFAR-10 and Tiny ImageNet
datasets show the effectiveness of our proposed attack in increasing energy
consumption on trigger samples while preserving the model's performance for
clean/regular inputs. This demonstrates the vulnerability of DNNs to energy
backdoor attacks. The source code of our attack is available at:
https://github.com/hbrachemi/energy_backdoor.


http://arxiv.org/abs/2501.07927
Gandalf the Red: Adaptive Security for LLMs. (38%)
Niklas Pfister; Václav Volhejn; Manuel Knott; Santiago Arias; Julia Bazińska; Mykhailo Bichurin; Alan Commike; Janet Darling; Peter Dienes; Matthew Fiedler; David Haber; Matthias Kraft; Marco Lancini; Max Mathys; Damián Pascual-Ortiz; Jakub Podolak; Adrià Romero-López; Kyriacos Shiarlis; Andreas Signer; Zsolt Terek; Athanasios Theocharis; Daniel Timbrell; Samuel Trautwein; Samuel Watts; Natalie Wu; Mateo Rojas-Carulla
  Current evaluations of defenses against prompt attacks in large language
model (LLM) applications often overlook two critical factors: the dynamic
nature of adversarial behavior and the usability penalties imposed on
legitimate users by restrictive defenses. We propose D-SEC (Dynamic Security
Utility Threat Model), which explicitly separates attackers from legitimate
users, models multi-step interactions, and rigorously expresses the
security-utility in an optimizable form. We further address the shortcomings in
existing evaluations by introducing Gandalf, a crowd-sourced, gamified
red-teaming platform designed to generate realistic, adaptive attack datasets.
Using Gandalf, we collect and release a dataset of 279k prompt attacks.
Complemented by benign user data, our analysis reveals the interplay between
security and utility, showing that defenses integrated in the LLM (e.g., system
prompts) can degrade usability even without blocking requests. We demonstrate
that restricted application domains, defense-in-depth, and adaptive defenses
are effective strategies for building secure and useful LLM applications. Code
is available at
\href{https://github.com/lakeraai/dsec-gandalf}{\texttt{https://github.com/lakeraai/dsec-gandalf}}.


http://arxiv.org/abs/2501.09039
Playing Devil's Advocate: Unmasking Toxicity and Vulnerabilities in Large Vision-Language Models. (2%)
Abdulkadir Erol; Trilok Padhi; Agnik Saha; Ugur Kursuncu; Mehmet Emin Aktas
  The rapid advancement of Large Vision-Language Models (LVLMs) has enhanced
capabilities offering potential applications from content creation to
productivity enhancement. Despite their innovative potential, LVLMs exhibit
vulnerabilities, especially in generating potentially toxic or unsafe
responses. Malicious actors can exploit these vulnerabilities to propagate
toxic content in an automated (or semi-) manner, leveraging the susceptibility
of LVLMs to deception via strategically crafted prompts without fine-tuning or
compute-intensive procedures. Despite the red-teaming efforts and inherent
potential risks associated with the LVLMs, exploring vulnerabilities of LVLMs
remains nascent and yet to be fully addressed in a systematic manner. This
study systematically examines the vulnerabilities of open-source LVLMs,
including LLaVA, InstructBLIP, Fuyu, and Qwen, using adversarial prompt
strategies that simulate real-world social manipulation tactics informed by
social theories. Our findings show that (i) toxicity and insulting are the most
prevalent behaviors, with the mean rates of 16.13% and 9.75%, respectively;
(ii) Qwen-VL-Chat, LLaVA-v1.6-Vicuna-7b, and InstructBLIP-Vicuna-7b are the
most vulnerable models, exhibiting toxic response rates of 21.50%, 18.30% and
17.90%, and insulting responses of 13.40%, 11.70% and 10.10%, respectively;
(iii) prompting strategies incorporating dark humor and multimodal toxic prompt
completion significantly elevated these vulnerabilities. Despite being
fine-tuned for safety, these models still generate content with varying degrees
of toxicity when prompted with adversarial inputs, highlighting the urgent need
for enhanced safety mechanisms and robust guardrails in LVLM development.


http://arxiv.org/abs/2501.07251
MOS-Attack: A Scalable Multi-objective Adversarial Attack Framework. (99%)
Ping Guo; Cheng Gong; Xi Lin; Fei Liu; Zhichao Lu; Qingfu Zhang; Zhenkun Wang
  Crafting adversarial examples is crucial for evaluating and enhancing the
robustness of Deep Neural Networks (DNNs), presenting a challenge equivalent to
maximizing a non-differentiable 0-1 loss function.
  However, existing single objective methods, namely adversarial attacks focus
on a surrogate loss function, do not fully harness the benefits of engaging
multiple loss functions, as a result of insufficient understanding of their
synergistic and conflicting nature.
  To overcome these limitations, we propose the Multi-Objective Set-based
Attack (MOS Attack), a novel adversarial attack framework leveraging multiple
loss functions and automatically uncovering their interrelations.
  The MOS Attack adopts a set-based multi-objective optimization strategy,
enabling the incorporation of numerous loss functions without additional
parameters.
  It also automatically mines synergistic patterns among various losses,
facilitating the generation of potent adversarial attacks with fewer
objectives.
  Extensive experiments have shown that our MOS Attack outperforms
single-objective attacks. Furthermore, by harnessing the identified synergistic
patterns, MOS Attack continues to show superior results with a reduced number
of loss functions.


http://arxiv.org/abs/2501.07493
Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards. (89%)
Yangsibo Huang; Milad Nasr; Anastasios Angelopoulos; Nicholas Carlini; Wei-Lin Chiang; Christopher A. Choquette-Choo; Daphne Ippolito; Matthew Jagielski; Katherine Lee; Ken Ziyu Liu; Ion Stoica; Florian Tramer; Chiyuan Zhang
  It is now common to evaluate Large Language Models (LLMs) by having humans
manually vote to evaluate model outputs, in contrast to typical benchmarks that
evaluate knowledge or skill at some particular task. Chatbot Arena, the most
popular benchmark of this type, ranks models by asking users to select the
better response between two randomly selected models (without revealing which
model was responsible for the generations). These platforms are widely trusted
as a fair and accurate measure of LLM capabilities. In this paper, we show that
if bot protection and other defenses are not implemented, these voting-based
benchmarks are potentially vulnerable to adversarial manipulation.
Specifically, we show that an attacker can alter the leaderboard (to promote
their favorite model or demote competitors) at the cost of roughly a thousand
votes (verified in a simulated, offline version of Chatbot Arena). Our attack
consists of two steps: first, we show how an attacker can determine which model
was used to generate a given reply with more than $95\%$ accuracy; and then,
the attacker can use this information to consistently vote for (or against) a
target model. Working with the Chatbot Arena developers, we identify, propose,
and implement mitigations to improve the robustness of Chatbot Arena against
adversarial manipulation, which, based on our analysis, substantially increases
the cost of such attacks. Some of these defenses were present before our
collaboration, such as bot protection with Cloudflare, malicious user
detection, and rate limiting. Others, including reCAPTCHA and login are being
integrated to strengthen the security in Chatbot Arena.


http://arxiv.org/abs/2501.07275
Generating Poisoning Attacks against Ridge Regression Models with Categorical Features. (64%)
Monse Guedes-Ayala; Lars Schewe; Zeynep Suvak; Miguel Anjos
  Machine Learning (ML) models have become a very powerful tool to extract
information from large datasets and use it to make accurate predictions and
automated decisions. However, ML models can be vulnerable to external attacks,
causing them to underperform or deviate from their expected tasks. One way to
attack ML models is by injecting malicious data to mislead the algorithm during
the training phase, which is referred to as a poisoning attack. We can prepare
for such situations by designing anticipated attacks, which are later used for
creating and testing defence strategies. In this paper, we propose an algorithm
to generate strong poisoning attacks for a ridge regression model containing
both numerical and categorical features that explicitly models and poisons
categorical features. We model categorical features as SOS-1 sets and formulate
the problem of designing poisoning attacks as a bilevel optimization problem
that is nonconvex mixed-integer in the upper-level and unconstrained convex
quadratic in the lower-level. We present the mathematical formulation of the
problem, introduce a single-level reformulation based on the Karush-Kuhn-Tucker
(KKT) conditions of the lower level, find bounds for the lower-level variables
to accelerate solver performance, and propose a new algorithm to poison
categorical features. Numerical experiments show that our method improves the
mean squared error of all datasets compared to the previous benchmark in the
literature.


http://arxiv.org/abs/2501.07192
A4O: All Trigger for One sample. (2%)
Duc Anh Vu; Anh Tuan Tran; Cong Tran; Cuong Pham
  Backdoor attacks have become a critical threat to deep neural networks
(DNNs), drawing many research interests. However, most of the studied attacks
employ a single type of trigger. Consequently, proposed backdoor defenders
often rely on the assumption that triggers would appear in a unified way. In
this paper, we show that this naive assumption can create a loophole, allowing
more sophisticated backdoor attacks to bypass. We design a novel backdoor
attack mechanism that incorporates multiple types of backdoor triggers,
focusing on stealthiness and effectiveness. Our journey begins with the
intriguing observation that the performance of a backdoor attack in deep
learning models, as well as its detectability and removability, are all
proportional to the magnitude of the trigger. Based on this correlation, we
propose reducing the magnitude of each trigger type and combining them to
achieve a strong backdoor relying on the combined trigger while still staying
safely under the radar of defenders. Extensive experiments on three standard
datasets demonstrate that our method can achieve high attack success rates
(ASRs) while consistently bypassing state-of-the-art defenses.


http://arxiv.org/abs/2501.07670
A Survey of Early Exit Deep Neural Networks in NLP. (1%)
Divya Jyoti Bajpai; Manjesh Kumar Hanawal
  Deep Neural Networks (DNNs) have grown increasingly large in size to achieve
state of the art performance across a wide range of tasks. However, their high
computational requirements make them less suitable for resource-constrained
applications. Also, real-world datasets often consist of a mixture of easy and
complex samples, necessitating adaptive inference mechanisms that account for
sample difficulty. Early exit strategies offer a promising solution by enabling
adaptive inference, where simpler samples are classified using the initial
layers of the DNN, thereby accelerating the overall inference process. By
attaching classifiers at different layers, early exit methods not only reduce
inference latency but also improve the model robustness against adversarial
attacks. This paper presents a comprehensive survey of early exit methods and
their applications in NLP.


http://arxiv.org/abs/2501.07752
Towards the Pseudorandomness of Expander Random Walks for Read-Once ACC0 circuits. (1%)
Emile Anand
  Expander graphs are among the most useful combinatorial objects in
theoretical computer science. A line of work studies random walks on expander
graphs for their pseudorandomness against various classes of test functions,
including symmetric functions, read-only branching programs, permutation
branching programs, and $\mathrm{AC}^0$ circuits. The promising results of
pseudorandomness of expander random walks against $\mathrm{AC}^0$ circuits
indicate a robustness of expander random walks beyond symmetric functions,
motivating the question of whether expander random walks can fool more robust
\emph{asymmetric} complexity classes, such as $\mathrm{ACC}^0$. In this work,
we make progress towards this question by considering certain two-layered
circuit compositions of $\mathrm{MOD}[k]$ gates, where we show that these
family of circuits are fooled by expander random walks with total variation
distance error $O(\lambda)$, where $\lambda$ is the second largest eigenvalue
of the underlying expander graph. For $k\geq 3$, these circuits can be highly
asymmetric with complicated Fourier characters. In this context, our work takes
a step in the direction of fooling more complex asymmetric circuits.
Separately, drawing from the learning-theory literature, we construct an
explicit threshold circuit in the circuit family $\mathrm{TC}^0$, and show that
it \emph{is} fooled by expander random walks, providing an upper bound on the
set of functions fooled by expander random walks.


http://arxiv.org/abs/2501.07044
Protego: Detecting Adversarial Examples for Vision Transformers via Intrinsic Capabilities. (99%)
Jialin Wu; Kaikai Pan; Yanjiao Chen; Jiangyi Deng; Shengyuan Pang; Wenyuan Xu
  Transformer models have excelled in natural language tasks, prompting the
vision community to explore their implementation in computer vision problems.
However, these models are still influenced by adversarial examples. In this
paper, we investigate the attack capabilities of six common adversarial attacks
on three pretrained ViT models to reveal the vulnerability of ViT models. To
understand and analyse the bias in neural network decisions when the input is
adversarial, we use two visualisation techniques that are attention rollout and
grad attention rollout. To prevent ViT models from adversarial attack, we
propose Protego, a detection framework that leverages the transformer intrinsic
capabilities to detection adversarial examples of ViT models. Nonetheless, this
is challenging due to a diversity of attack strategies that may be adopted by
adversaries. Inspired by the attention mechanism, we know that the token of
prediction contains all the information from the input sample. Additionally,
the attention region for adversarial examples differs from that of normal
examples. Given these points, we can train a detector that achieves superior
performance than existing detection methods to identify adversarial examples.
Our experiments have demonstrated the high effectiveness of our detection
method. For these six adversarial attack methods, our detector's AUC scores all
exceed 0.95. Protego may advance investigations in metaverse security.


http://arxiv.org/abs/2501.06729
KeTS: Kernel-based Trust Segmentation against Model Poisoning Attacks. (98%)
Ankit Gangwal; Mauro Conti; Tommaso Pauselli
  Federated Learning (FL) enables multiple users to collaboratively train a
global model in a distributed manner without revealing their personal data.
However, FL remains vulnerable to model poisoning attacks, where malicious
actors inject crafted updates to compromise the global model's accuracy. These
vulnerabilities are particularly severe in non-homogeneous environments, where
clients exhibit varying proportions of class labels, resulting in heterogeneous
updates. In such settings, benign outliers are often misclassified as false
positives, while maliciously crafted uploads evade detection and are aggregated
at the server. Existing defense mechanisms struggle in such real-world
settings, resulting in significant declines in the global FL model's
performance.
  We propose a novel defense mechanism, Kernel-based Trust Segmentation (KeTS),
to counter model poisoning attacks. Unlike existing approaches, KeTS analyzes
the evolution of each client's updates and effectively segments malicious
clients using Kernel Density Estimation (KDE), even in the presence of benign
outliers. We thoroughly evaluate KeTS's performance against the six most
effective model poisoning attacks (i.e., Trim-Attack, Krum-Attack, Min-Max
attack, Min-Sum attack, and their variants) on two different datasets (i.e.,
MNIST and Fashion-MNIST) and compare its performance with three classical
robust schemes (i.e., Krum, Trim-Mean, and Median) and a state-of-the-art
defense (i.e., FLTrust). Our results show that KeTS outperforms the existing
defenses in every attack setting; beating the best-performing defense by an
overall average of >24% (on MNIST) and >14% (on Fashion-MNIST). A series of
further experiments (varying poisoning approaches, attacker population, etc.)
reveal the consistent and superior performance of KeTS under diverse
conditions.


http://arxiv.org/abs/2501.06736
ZOQO: Zero-Order Quantized Optimization. (1%)
Noga Bar; Raja Giryes
  The increasing computational and memory demands in deep learning present
significant challenges, especially in resource-constrained environments. We
introduce a zero-order quantized optimization (ZOQO) method designed for
training models with quantized parameters and operations. Our approach
leverages zero-order approximations of the gradient sign and adapts the
learning process to maintain the parameters' quantization without the need for
full-precision gradient calculations. We demonstrate the effectiveness of ZOQO
through experiments in fine-tuning of large language models and black-box
adversarial attacks. Despite the limitations of zero-order and quantized
operations training, our method achieves competitive performance compared to
full-precision methods, highlighting its potential for low-resource
environments.


http://arxiv.org/abs/2501.06646
RogueRFM: Attacking Refresh Management for Covert-Channel and Denial-of-Service. (10%)
Hritvik Taneja; Moinuddin Qureshi
  With lowering thresholds, transparently defending against Rowhammer within
DRAM is challenging due to the lack of time to perform mitigation. Commercially
deployed in-DRAM defenses like TRR that steal time from normal refreshes~(REF)
to perform mitigation have been proven ineffective against Rowhammer. In
response, a new Refresh Management (RFM) interface has been added to the DDR5
specifications. RFM provides dedicated time to an in-DRAM defense to perform
mitigation. Several recent works have used RFM for the intended purpose -
building better Rowhammer defenses. However, to the best of our knowledge, no
prior study has looked at the potential security implications of this new
feature if an attacker subjects it to intentional misuse.
  Our paper shows that RFM introduces new side effects in the system - the
activity of one bank causes interference with the operation of the other banks.
Thus, the latency of a bank becomes dependent on the activity of other banks.
We use these side effects to build two new attacks. First, a novel memory-based
covert channel, which has a bandwidth of up to 31.3 KB/s, and is also effective
even in a bank-partitioned system. Second, a new Denial-of-Service (DOS) attack
pattern that exploits the activity within a single bank to reduce the
performance of the other banks. Our experiments on SPEC2017, PARSEC, and LIGRA
workloads show a slowdown of up to 67\% when running alongside our DOS pattern.
We also discuss potential countermeasures for our attacks.


http://arxiv.org/abs/2501.06650
SafeSplit: A Novel Defense Against Client-Side Backdoor Attacks in Split Learning. (10%)
Phillip Rieger; Alessandro Pegoraro; Kavita Kumari; Tigist Abera; Jonathan Knauer; Ahmad-Reza Sadeghi
  Split Learning (SL) is a distributed deep learning approach enabling multiple
clients and a server to collaboratively train and infer on a shared deep neural
network (DNN) without requiring clients to share their private local data. The
DNN is partitioned in SL, with most layers residing on the server and a few
initial layers and inputs on the client side. This configuration allows
resource-constrained clients to participate in training and inference. However,
the distributed architecture exposes SL to backdoor attacks, where malicious
clients can manipulate local datasets to alter the DNN's behavior. Existing
defenses from other distributed frameworks like Federated Learning are not
applicable, and there is a lack of effective backdoor defenses specifically
designed for SL.
  We present SafeSplit, the first defense against client-side backdoor attacks
in Split Learning (SL). SafeSplit enables the server to detect and filter out
malicious client behavior by employing circular backward analysis after a
client's training is completed, iteratively reverting to a trained checkpoint
where the model under examination is found to be benign. It uses a two-fold
analysis to identify client-induced changes and detect poisoned models. First,
a static analysis in the frequency domain measures the differences in the
layer's parameters at the server. Second, a dynamic analysis introduces a novel
rotational distance metric that assesses the orientation shifts of the server's
layer parameters during training. Our comprehensive evaluation across various
data distributions, client counts, and attack scenarios demonstrates the high
efficacy of this dual analysis in mitigating backdoor attacks while preserving
model utility.


http://arxiv.org/abs/2501.05962
Effective faking of verbal deception detection with target-aligned adversarial attacks. (87%)
Bennett Kleinberg; Riccardo Loconte; Bruno Verschuere
  Background: Deception detection through analysing language is a promising
avenue using both human judgments and automated machine learning judgments. For
both forms of credibility assessment, automated adversarial attacks that
rewrite deceptive statements to appear truthful pose a serious threat. Methods:
We used a dataset of 243 truthful and 262 fabricated autobiographical stories
in a deception detection task for humans and machine learning models. A large
language model was tasked to rewrite deceptive statements so that they appear
truthful. In Study 1, humans who made a deception judgment or used the
detailedness heuristic and two machine learning models (a fine-tuned language
model and a simple n-gram model) judged original or adversarial modifications
of deceptive statements. In Study 2, we manipulated the target alignment of the
modifications, i.e. tailoring the attack to whether the statements would be
assessed by humans or computer models. Results: When adversarial modifications
were aligned with their target, human (d=-0.07 and d=-0.04) and machine
judgments (51% accuracy) dropped to the chance level. When the attack was not
aligned with the target, both human heuristics judgments (d=0.30 and d=0.36)
and machine learning predictions (63-78%) were significantly better than
chance. Conclusions: Easily accessible language models can effectively help
anyone fake deception detection efforts both by humans and machine learning
models. Robustness against adversarial modifications for humans and machines
depends on that target alignment. We close with suggestions on advancing
deception research with adversarial attack designs.


http://arxiv.org/abs/2501.05783
UV-Attack: Physical-World Adversarial Attacks for Person Detection via Dynamic-NeRF-based UV Mapping. (81%)
Yanjie Li; Wenxuan Zhang; Kaisheng Liang; Bin Xiao
  In recent research, adversarial attacks on person detectors using patches or
static 3D model-based texture modifications have struggled with low success
rates due to the flexible nature of human movement. Modeling the 3D
deformations caused by various actions has been a major challenge. Fortunately,
advancements in Neural Radiance Fields (NeRF) for dynamic human modeling offer
new possibilities. In this paper, we introduce UV-Attack, a groundbreaking
approach that achieves high success rates even with extensive and unseen human
actions. We address the challenge above by leveraging dynamic-NeRF-based UV
mapping. UV-Attack can generate human images across diverse actions and
viewpoints, and even create novel actions by sampling from the SMPL parameter
space. While dynamic NeRF models are capable of modeling human bodies,
modifying clothing textures is challenging because they are embedded in neural
network parameters. To tackle this, UV-Attack generates UV maps instead of RGB
images and modifies the texture stacks. This approach enables real-time texture
edits and makes the attack more practical. We also propose a novel Expectation
over Pose Transformation loss (EoPT) to improve the evasion success rate on
unseen poses and views. Our experiments show that UV-Attack achieves a 92.75%
attack success rate against the FastRCNN model across varied poses in dynamic
video settings, significantly outperforming the state-of-the-art AdvCamou
attack, which only had a 28.50% ASR. Moreover, we achieve 49.5% ASR on the
latest YOLOv8 detector in black-box settings. This work highlights the
potential of dynamic NeRF-based UV mapping for creating more effective
adversarial attacks on person detectors, addressing key challenges in modeling
human movement and texture modification.


http://arxiv.org/abs/2501.05835
Fine-tuning is Not Fine: Mitigating Backdoor Attacks in GNNs with Limited Clean Data. (80%)
Jiale Zhang; Bosen Rao; Chengcheng Zhu; Xiaobing Sun; Qingming Li; Haibo Hu; Xiapu Luo; Qingqing Ye; Shouling Ji
  Graph Neural Networks (GNNs) have achieved remarkable performance through
their message-passing mechanism. However, recent studies have highlighted the
vulnerability of GNNs to backdoor attacks, which can lead the model to
misclassify graphs with attached triggers as the target class. The
effectiveness of recent promising defense techniques, such as fine-tuning or
distillation, is heavily contingent on having comprehensive knowledge of the
sufficient training dataset. Empirical studies have shown that fine-tuning
methods require a clean dataset of 20% to reduce attack accuracy to below 25%,
while distillation methods require a clean dataset of 15%. However, obtaining
such a large amount of clean data is commonly impractical.
  In this paper, we propose a practical backdoor mitigation framework, denoted
as GRAPHNAD, which can capture high-quality intermediate-layer representations
in GNNs to enhance the distillation process with limited clean data. To achieve
this, we address the following key questions: How to identify the appropriate
attention representations in graphs for distillation? How to enhance
distillation with limited data? By adopting the graph attention transfer
method, GRAPHNAD can effectively align the intermediate-layer attention
representations of the backdoored model with that of the teacher model, forcing
the backdoor neurons to transform into benign ones. Besides, we extract the
relation maps from intermediate-layer transformation and enforce the relation
maps of the backdoored model to be consistent with that of the teacher model,
thereby ensuring model accuracy while further reducing the influence of
backdoors. Extensive experimental results show that by fine-tuning a teacher
model with only 3% of the clean data, GRAPHNAD can reduce the attack success
rate to below 5%.


http://arxiv.org/abs/2501.05928
Towards Backdoor Stealthiness in Model Parameter Space. (61%)
Xiaoyun Xu; Zhuoran Liu; Stefanos Koffas; Stjepan Picek
  Recent research on backdoor stealthiness focuses mainly on indistinguishable
triggers in input space and inseparable backdoor representations in feature
space, aiming to circumvent backdoor defenses that examine these respective
spaces. However, existing backdoor attacks are typically designed to resist a
specific type of backdoor defense without considering the diverse range of
defense mechanisms. Based on this observation, we pose a natural question: Are
current backdoor attacks truly a real-world threat when facing diverse
practical defenses?
  To answer this question, we examine 12 common backdoor attacks that focus on
input-space or feature-space stealthiness and 17 diverse representative
defenses. Surprisingly, we reveal a critical blind spot: Backdoor attacks
designed to be stealthy in input and feature spaces can be mitigated by
examining backdoored models in parameter space. To investigate the underlying
causes behind this common vulnerability, we study the characteristics of
backdoor attacks in the parameter space. Notably, we find that input- and
feature-space attacks introduce prominent backdoor-related neurons in parameter
space, which are not thoroughly considered by current backdoor attacks. Taking
comprehensive stealthiness into account, we propose a novel supply-chain attack
called Grond. Grond limits the parameter changes by a simple yet effective
module, Adversarial Backdoor Injection (ABI), which adaptively increases the
parameter-space stealthiness during the backdoor injection. Extensive
experiments demonstrate that Grond outperforms all 12 backdoor attacks against
state-of-the-art (including adaptive) defenses on CIFAR-10, GTSRB, and a subset
of ImageNet. In addition, we show that ABI consistently improves the
effectiveness of common backdoor attacks.


http://arxiv.org/abs/2501.05991
An Attention-Guided Deep Learning Approach for Classifying 39 Skin Lesion Types. (1%)
Sauda Adiv Hanum; Ashim Dey; Muhammad Ashad Kabir
  The skin, as the largest organ of the human body, is vulnerable to a diverse
array of conditions collectively known as skin lesions, which encompass various
dermatoses. Diagnosing these lesions presents significant challenges for
medical practitioners due to the subtle visual differences that are often
imperceptible to the naked eye. While not all skin lesions are
life-threatening, certain types can act as early indicators of severe diseases,
including skin cancers, underscoring the critical need for timely and accurate
diagnostic methods. Deep learning algorithms have demonstrated remarkable
potential in facilitating the early detection and prognosis of skin lesions.
This study advances the field by curating a comprehensive and diverse dataset
comprising 39 categories of skin lesions, synthesized from five publicly
available datasets. Using this dataset, the performance of five
state-of-the-art deep learning models -- MobileNetV2, Xception, InceptionV3,
EfficientNetB1, and Vision Transformer - is rigorously evaluated. To enhance
the accuracy and robustness of these models, attention mechanisms such as the
Efficient Channel Attention (ECA) and the Convolutional Block Attention Module
(CBAM) are incorporated into their architectures. Comprehensive evaluation
across multiple performance metrics reveals that the Vision Transformer model
integrated with CBAM outperforms others, achieving an accuracy of 93.46%,
precision of 94%, recall of 93%, F1-score of 93%, and specificity of 93.67%.
These results underscore the significant potential of the proposed system in
supporting medical professionals with accurate and efficient prognostic tools
for diagnosing a broad spectrum of skin lesions. The dataset and code used in
this study can be found at
https://github.com/akabircs/Skin-Lesions-Classification.


http://arxiv.org/abs/2501.06264
HPAC-IDS: A Hierarchical Packet Attention Convolution for Intrusion Detection System. (99%)
Anass Grini; Btissam El Khamlichi; Abdellatif El Afia; Amal El Fallah-Seghrouchni
  This research introduces a robust detection system against malicious network
traffic, leveraging hierarchical structures and self-attention mechanisms. The
proposed system includes a Packet Segmenter that divides a given raw network
packet into fixed-size segments that are fed to the HPAC-IDS. The experiments
performed on CIC-IDS2017 dataset show that the system exhibits high accuracy
and low false positive rates while demonstrating resilience against diverse
adversarial methods like Fast Gradient Sign Method (FGSM), Projected Gradient
Descent (PGD), and Wasserstein GAN (WGAN). The model's ability to withstand
adversarial perturbations is attributed to the fusion of hierarchical attention
mechanisms and convolutional neural networks, resulting in a 0% to 10%
adversarial attack severity under tested adversarial attacks with different
segment sizes, surpassing the state-of-the-art model in detection performance
and adversarial attack robustness.


http://arxiv.org/abs/2501.05127
DiffAttack: Diffusion-based Timbre-reserved Adversarial Attack in Speaker Identification. (97%)
Qing Wang; Jixun Yao; Zhaokai Sun; Pengcheng Guo; Lei Xie; John H. L. Hansen
  Being a form of biometric identification, the security of the speaker
identification (SID) system is of utmost importance. To better understand the
robustness of SID systems, we aim to perform more realistic attacks in SID,
which are challenging for both humans and machines to detect. In this study, we
propose DiffAttack, a novel timbre-reserved adversarial attack approach that
exploits the capability of a diffusion-based voice conversion (DiffVC) model to
generate adversarial fake audio with distinct target speaker attribution. By
introducing adversarial constraints into the generative process of the
diffusion-based voice conversion model, we craft fake samples that effectively
mislead target models while preserving speaker-wise characteristics.
Specifically, inspired by the use of randomly sampled Gaussian noise in
conventional adversarial attacks and diffusion processes, we incorporate
adversarial constraints into the reverse diffusion process. These constraints
subtly guide the reverse diffusion process toward aligning with the target
speaker distribution. Our experiments on the LibriTTS dataset indicate that
DiffAttack significantly improves the attack success rate compared to vanilla
DiffVC and other methods. Moreover, objective and subjective evaluations
demonstrate that introducing adversarial constraints does not compromise the
speech quality generated by the DiffVC model.


http://arxiv.org/abs/2501.05588
Enforcing Fundamental Relations via Adversarial Attacks on Input Parameter Correlations. (92%)
Timo Saala; Lucie Flek; Alexander Jung; Akbar Karimi; Alexander Schmidt; Matthias Schott; Philipp Soldin; Christopher Wiebusch
  Correlations between input parameters play a crucial role in many scientific
classification tasks, since these are often related to fundamental laws of
nature. For example, in high energy physics, one of the common deep learning
use-cases is the classification of signal and background processes in particle
collisions. In many such cases, the fundamental principles of the correlations
between observables are often better understood than the actual distributions
of the observables themselves. In this work, we present a new adversarial
attack algorithm called Random Distribution Shuffle Attack (RDSA), emphasizing
the correlations between observables in the network rather than individual
feature characteristics. Correct application of the proposed novel attack can
result in a significant improvement in classification performance -
particularly in the context of data augmentation - when using the generated
adversaries within adversarial training. Given that correlations between input
features are also crucial in many other disciplines. We demonstrate the RDSA
effectiveness on six classification tasks, including two particle collision
challenges (using CERN Open Data), hand-written digit recognition (MNIST784),
human activity recognition (HAR), weather forecasting (Rain in Australia), and
ICU patient mortality (MIMIC-IV), demonstrating a general use case beyond
fundamental physics for this new type of adversarial attack algorithms.


http://arxiv.org/abs/2501.04985
SpaLLM-Guard: Pairing SMS Spam Detection Using Open-source and Commercial LLMs. (50%)
Muhammad Salman; Muhammad Ikram; Nardine Basta; Mohamed Ali Kaafar
  The increasing threat of SMS spam, driven by evolving adversarial techniques
and concept drift, calls for more robust and adaptive detection methods. In
this paper, we evaluate the potential of large language models (LLMs), both
open-source and commercial, for SMS spam detection, comparing their performance
across zero-shot, few-shot, fine-tuning, and chain-of-thought prompting
approaches. Using a comprehensive dataset of SMS messages, we assess the spam
detection capabilities of prominent LLMs such as GPT-4, DeepSeek, LLAMA-2, and
Mixtral. Our findings reveal that while zero-shot learning provides
convenience, it is unreliable for effective spam detection. Few-shot learning,
particularly with carefully selected examples, improves detection but exhibits
variability across models. Fine-tuning emerges as the most effective strategy,
with Mixtral achieving 98.6% accuracy and a balanced false positive and false
negative rate below 2%, meeting the criteria for robust spam detection.
Furthermore, we explore the resilience of these models to adversarial attacks,
finding that fine-tuning significantly enhances robustness against both
perceptible and imperceptible manipulations. Lastly, we investigate the impact
of concept drift and demonstrate that fine-tuned LLMs, especially when combined
with few-shot learning, can mitigate its effects, maintaining high performance
even on evolving spam datasets. This study highlights the importance of
fine-tuning and tailored learning strategies to deploy LLMs effectively for
real-world SMS spam detection


http://arxiv.org/abs/2501.05015
On Measuring Unnoticeability of Graph Adversarial Attacks: Observations, New Measure, and Applications. (33%)
Hyeonsoo Jo; Hyunjin Hwang; Fanchen Bu; Soo Yong Lee; Chanyoung Park; Kijung Shin
  Adversarial attacks are allegedly unnoticeable. Prior studies have designed
attack noticeability measures on graphs, primarily using statistical tests to
compare the topology of original and (possibly) attacked graphs. However, we
observe two critical limitations in the existing measures. First, because the
measures rely on simple rules, attackers can readily enhance their attacks to
bypass them, reducing their attack "noticeability" and, yet, maintaining their
attack performance. Second, because the measures naively leverage global
statistics, such as degree distributions, they may entirely overlook attacks
until severe perturbations occur, letting the attacks be almost "totally
unnoticeable." To address the limitations, we introduce HideNSeek, a learnable
measure for graph attack noticeability. First, to mitigate the bypass problem,
HideNSeek learns to distinguish the original and (potential) attack edges using
a learnable edge scorer (LEO), which scores each edge on its likelihood of
being an attack. Second, to mitigate the overlooking problem, HideNSeek
conducts imbalance-aware aggregation of all the edge scores to obtain the final
noticeability score. Using six real-world graphs, we empirically demonstrate
that HideNSeek effectively alleviates the observed limitations, and LEO (i.e.,
our learnable edge scorer) outperforms eleven competitors in distinguishing
attack edges under five different attack methods. For an additional
application, we show that LEO boost the performance of robust GNNs by removing
attack-like edges.


http://arxiv.org/abs/2501.05359
CROPS: Model-Agnostic Training-Free Framework for Safe Image Synthesis with Latent Diffusion Models. (13%)
Junha Park; Ian Ryu; Jaehui Hwang; Hyungkeun Park; Jiyoon Kim; Jong-Seok Lee
  With advances in diffusion models, image generation has shown significant
performance improvements. This raises concerns about the potential abuse of
image generation, such as the creation of explicit or violent images, commonly
referred to as Not Safe For Work (NSFW) content. To address this, the Stable
Diffusion model includes several safety checkers to censor initial text prompts
and final output images generated from the model. However, recent research has
shown that these safety checkers have vulnerabilities against adversarial
attacks, allowing them to generate NSFW images. In this paper, we find that
these adversarial attacks are not robust to small changes in text prompts or
input latents. Based on this, we propose CROPS (Circular or RandOm Prompts for
Safety), a model-agnostic framework that easily defends against adversarial
attacks generating NSFW images without requiring additional training. Moreover,
we develop an approach that utilizes one-step diffusion models for efficient
NSFW detection (CROPS-1), further reducing computational resources. We
demonstrate the superiority of our method in terms of performance and
applicability.


http://arxiv.org/abs/2501.05168
KabaddiPy: A package to enable access to Professional Kabaddi Data. (1%)
Bhaskar Lalwani; Aniruddha Mukherjee
  Kabaddi, a contact team sport of Indian origin, has seen a dramatic rise in
global popularity, highlighted by the upcoming Kabaddi World Cup in 2025 with
over sixteen international teams participating, alongside flourishing national
leagues such as the Indian Pro Kabaddi League (230 million viewers) and the
British Kabaddi League. We present the first open-source Python module to make
Kabaddi statistical data easily accessible from multiple scattered sources
across the internet. The module was developed by systematically web-scraping
and collecting team-wise, player-wise and match-by-match data. The data has
been cleaned, organized, and categorized into team overviews and player
metrics, each filterable by season. The players are classified as raiders and
defenders, with their best strategies for attacking, counter-attacking, and
defending against different teams highlighted. Our module enables continuous
monitoring of exponentially growing data streams, aiding researchers to quickly
start building upon the data to answer critical questions, such as the impact
of player inclusion/exclusion on team performance, scoring patterns against
specific teams, and break down opponent gameplay. The data generated from
Kabaddi tournaments has been sparsely used, and coaches and players rely
heavily on intuition to make decisions and craft strategies. Our module can be
utilized to build predictive models, craft uniquely strategic gameplays to
target opponents and identify hidden correlations in the data. This open source
module has the potential to increase time-efficiency, encourage analytical
studies of Kabaddi gameplay and player dynamics and foster reproducible
research. The data and code are publicly available:
https://github.com/kabaddiPy/kabaddiPy


http://arxiv.org/abs/2501.05239
Is Your Autonomous Vehicle Safe? Understanding the Threat of Electromagnetic Signal Injection Attacks on Traffic Scene Perception. (1%)
Wenhao Liao; Sineng Yan; Youqian Zhang; Xinwei Zhai; Yuanyuan Wang; Eugene Yujun Fu
  Autonomous vehicles rely on camera-based perception systems to comprehend
their driving environment and make crucial decisions, thereby ensuring vehicles
to steer safely. However, a significant threat known as Electromagnetic Signal
Injection Attacks (ESIA) can distort the images captured by these cameras,
leading to incorrect AI decisions and potentially compromising the safety of
autonomous vehicles. Despite the serious implications of ESIA, there is limited
understanding of its impacts on the robustness of AI models across various and
complex driving scenarios. To address this gap, our research analyzes the
performance of different models under ESIA, revealing their vulnerabilities to
the attacks. Moreover, due to the challenges in obtaining real-world attack
data, we develop a novel ESIA simulation method and generate a simulated attack
dataset for different driving scenarios. Our research provides a comprehensive
simulation and evaluation framework, aiming to enhance the development of more
robust AI models and secure intelligent systems, ultimately contributing to the
advancement of safer and more reliable technology across various fields.


http://arxiv.org/abs/2501.05249
RAG-WM: An Efficient Black-Box Watermarking Approach for Retrieval-Augmented Generation of Large Language Models. (1%)
Peizhuo Lv; Mengjie Sun; Hao Wang; Xiaofeng Wang; Shengzhi Zhang; Yuxuan Chen; Kai Chen; Limin Sun
  In recent years, tremendous success has been witnessed in Retrieval-Augmented
Generation (RAG), widely used to enhance Large Language Models (LLMs) in
domain-specific, knowledge-intensive, and privacy-sensitive tasks. However,
attackers may steal those valuable RAGs and deploy or commercialize them,
making it essential to detect Intellectual Property (IP) infringement. Most
existing ownership protection solutions, such as watermarks, are designed for
relational databases and texts. They cannot be directly applied to RAGs because
relational database watermarks require white-box access to detect IP
infringement, which is unrealistic for the knowledge base in RAGs. Meanwhile,
post-processing by the adversary's deployed LLMs typically destructs text
watermark information. To address those problems, we propose a novel black-box
"knowledge watermark" approach, named RAG-WM, to detect IP infringement of
RAGs. RAG-WM uses a multi-LLM interaction framework, comprising a Watermark
Generator, Shadow LLM & RAG, and Watermark Discriminator, to create watermark
texts based on watermark entity-relationship tuples and inject them into the
target RAG. We evaluate RAG-WM across three domain-specific and two
privacy-sensitive tasks on four benchmark LLMs. Experimental results show that
RAG-WM effectively detects the stolen RAGs in various deployed LLMs.
Furthermore, RAG-WM is robust against paraphrasing, unrelated content removal,
knowledge insertion, and knowledge expansion attacks. Lastly, RAG-WM can also
evade watermark detection approaches, highlighting its promising application in
detecting IP infringement of RAG systems.


http://arxiv.org/abs/2501.04861
LayerMix: Enhanced Data Augmentation through Fractal Integration for Robust Deep Learning. (86%)
Hafiz Mughees Ahmad; Dario Morle; Afshin Rahimi
  Deep learning models have demonstrated remarkable performance across various
computer vision tasks, yet their vulnerability to distribution shifts remains a
critical challenge. Despite sophisticated neural network architectures,
existing models often struggle to maintain consistent performance when
confronted with Out-of-Distribution (OOD) samples, including natural
corruptions, adversarial perturbations, and anomalous patterns. We introduce
LayerMix, an innovative data augmentation approach that systematically enhances
model robustness through structured fractal-based image synthesis. By
meticulously integrating structural complexity into training datasets, our
method generates semantically consistent synthetic samples that significantly
improve neural network generalization capabilities. Unlike traditional
augmentation techniques that rely on random transformations, LayerMix employs a
structured mixing pipeline that preserves original image semantics while
introducing controlled variability. Extensive experiments across multiple
benchmark datasets, including CIFAR-10, CIFAR-100, ImageNet-200, and
ImageNet-1K demonstrate LayerMixs superior performance in classification
accuracy and substantially enhances critical Machine Learning (ML) safety
metrics, including resilience to natural image corruptions, robustness against
adversarial attacks, improved model calibration and enhanced prediction
consistency. LayerMix represents a significant advancement toward developing
more reliable and adaptable artificial intelligence systems by addressing the
fundamental challenges of deep learning generalization. The code is available
at https://github.com/ahmadmughees/layermix.


http://arxiv.org/abs/2501.04802
Reproducing HotFlip for Corpus Poisoning Attacks in Dense Retrieval. (84%)
Yongkang Li; Panagiotis Eustratiadis; Evangelos Kanoulas
  HotFlip is a topical gradient-based word substitution method for attacking
language models. Recently, this method has been further applied to attack
retrieval systems by generating malicious passages that are injected into a
corpus, i.e., corpus poisoning. However, HotFlip is known to be computationally
inefficient, with the majority of time being spent on gradient accumulation for
each query-passage pair during the adversarial token generation phase, making
it impossible to generate an adequate number of adversarial passages in a
reasonable amount of time. Moreover, the attack method itself assumes access to
a set of user queries, a strong assumption that does not correspond to how
real-world adversarial attacks are usually performed. In this paper, we first
significantly boost the efficiency of HotFlip, reducing the adversarial
generation process from 4 hours per document to only 15 minutes, using the same
hardware. We further contribute experiments and analysis on two additional
tasks: (1) transfer-based black-box attacks, and (2) query-agnostic attacks.
Whenever possible, we provide comparisons between the original method and our
improved version. Our experiments demonstrate that HotFlip can effectively
attack a variety of dense retrievers, with an observed trend that its attack
performance diminishes against more advanced and recent methods. Interestingly,
we observe that while HotFlip performs poorly in a black-box setting,
indicating limited capacity for generalization, in query-agnostic scenarios its
performance is correlated to the volume of injected adversarial passages.


http://arxiv.org/abs/2501.04453
Gradient Purification: Defense Against Poisoning Attack in Decentralized Federated Learning. (78%)
Bin Li; Xiaoye Miao; Yongheng Shang; Xinkui Zhao; Shuiguang Deng; Jianwei Yin
  Decentralized federated learning (DFL) is inherently vulnerable to poisoning
attacks, as malicious clients can transmit manipulated model gradients to
neighboring clients. Existing defense methods either reject suspicious
gradients per iteration or restart DFL aggregation after detecting all
malicious clients. They overlook the potential accuracy benefit from the
discarded malicious gradients. In this paper, we propose a novel gradient
purification defense, named GPD, that integrates seamlessly with existing DFL
aggregation to defend against poisoning attacks. It aims to mitigate the harm
in model gradients while retaining the benefit in model weights for enhancing
accuracy. For each benign client in GPD, a recording variable is designed to
track the historically aggregated gradients from one of its neighbors. It
allows benign clients to precisely detect malicious neighbors and swiftly
mitigate aggregated malicious gradients via historical consistency checks. Upon
mitigation, GPD optimizes model weights via aggregating gradients solely from
benign clients. This retains the previously beneficial portions from malicious
clients and exploits the contributions from benign clients, thereby
significantly enhancing the model accuracy. We analyze the convergence of GPD,
as well as its ability to harvest high accuracy. Extensive experiments over
three datasets demonstrate that, GPD is capable of mitigating poisoning attacks
under both iid and non-iid data distributions. It significantly outperforms
state-of-the-art defenses in terms of accuracy against various poisoning
attacks.


http://arxiv.org/abs/2501.04931
Jailbreaking Multimodal Large Language Models via Shuffle Inconsistency. (38%)
Shiji Zhao; Ranjie Duan; Fengxiang Wang; Chi Chen; Caixin Kang; Jialing Tao; YueFeng Chen; Hui Xue; Xingxing Wei
  Multimodal Large Language Models (MLLMs) have achieved impressive performance
and have been put into practical use in commercial applications, but they still
have potential safety mechanism vulnerabilities. Jailbreak attacks are red
teaming methods that aim to bypass safety mechanisms and discover MLLMs'
potential risks. Existing MLLMs' jailbreak methods often bypass the model's
safety mechanism through complex optimization methods or carefully designed
image and text prompts. Despite achieving some progress, they have a low attack
success rate on commercial closed-source MLLMs. Unlike previous research, we
empirically find that there exists a Shuffle Inconsistency between MLLMs'
comprehension ability and safety ability for the shuffled harmful instruction.
That is, from the perspective of comprehension ability, MLLMs can understand
the shuffled harmful text-image instructions well. However, they can be easily
bypassed by the shuffled harmful instructions from the perspective of safety
ability, leading to harmful responses. Then we innovatively propose a
text-image jailbreak attack named SI-Attack. Specifically, to fully utilize the
Shuffle Inconsistency and overcome the shuffle randomness, we apply a
query-based black-box optimization method to select the most harmful shuffled
inputs based on the feedback of the toxic judge model. A series of experiments
show that SI-Attack can improve the attack's performance on three benchmarks.
In particular, SI-Attack can obviously improve the attack success rate for
commercial MLLMs such as GPT-4o or Claude-3.5-Sonnet.


http://arxiv.org/abs/2501.04527
Towards Fair Class-wise Robustness: Class Optimal Distribution Adversarial Training. (26%)
Hongxin Zhi; Hongtao Yu; Shaome Li; Xiuming Zhao; Yiteng Wu
  Adversarial training has proven to be a highly effective method for improving
the robustness of deep neural networks against adversarial attacks.
Nonetheless, it has been observed to exhibit a limitation in terms of robust
fairness, characterized by a significant disparity in robustness across
different classes. Recent efforts to mitigate this problem have turned to
class-wise reweighted methods. However, these methods suffer from a lack of
rigorous theoretical analysis and are limited in their exploration of the
weight space, as they mainly rely on existing heuristic algorithms or intuition
to compute weights. In addition, these methods fail to guarantee the
consistency of the optimization direction due to the decoupled optimization of
weights and the model parameters. They potentially lead to suboptimal weight
assignments and consequently, a suboptimal model. To address these problems,
this paper proposes a novel min-max training framework, Class Optimal
Distribution Adversarial Training (CODAT), which employs distributionally
robust optimization to fully explore the class-wise weight space, thus enabling
the identification of the optimal weight with theoretical guarantees.
Furthermore, we derive a closed-form optimal solution to the internal
maximization and then get a deterministic equivalent objective function, which
provides a theoretical basis for the joint optimization of weights and model
parameters. Meanwhile, we propose a fairness elasticity coefficient for the
evaluation of the algorithm with regard to both robustness and robust fairness.
Experimental results on various datasets show that the proposed method can
effectively improve the robust fairness of the model and outperform the
state-of-the-art approaches.


http://arxiv.org/abs/2501.03562
Rethinking Adversarial Attacks in Reinforcement Learning from Policy Distribution Perspective. (81%)
Tianyang Duan; Zongyuan Zhang; Zheng Lin; Yue Gao; Ling Xiong; Yong Cui; Hongbin Liang; Xianhao Chen; Heming Cui; Dong Huang
  Deep Reinforcement Learning (DRL) suffers from uncertainties and inaccuracies
in the observation signal in realworld applications. Adversarial attack is an
effective method for evaluating the robustness of DRL agents. However, existing
attack methods targeting individual sampled actions have limited impacts on the
overall policy distribution, particularly in continuous action spaces. To
address these limitations, we propose the Distribution-Aware Projected Gradient
Descent attack (DAPGD). DAPGD uses distribution similarity as the gradient
perturbation input to attack the policy network, which leverages the entire
policy distribution rather than relying on individual samples. We utilize the
Bhattacharyya distance in DAPGD to measure policy similarity, enabling
sensitive detection of subtle but critical differences between probability
distributions. Our experiment results demonstrate that DAPGD achieves SOTA
results compared to the baselines in three robot navigation tasks, achieving an
average 22.03% higher reward drop compared to the best baseline.


http://arxiv.org/abs/2501.03940
Not all tokens are created equal: Perplexity Attention Weighted Networks for AI generated text detection. (3%)
Pablo Miralles-González; Javier Huertas-Tato; Alejandro Martín; David Camacho
  The rapid advancement in large language models (LLMs) has significantly
enhanced their ability to generate coherent and contextually relevant text,
raising concerns about the misuse of AI-generated content and making it
critical to detect it. However, the task remains challenging, particularly in
unseen domains or with unfamiliar LLMs. Leveraging LLM next-token distribution
outputs offers a theoretically appealing approach for detection, as they
encapsulate insights from the models' extensive pre-training on diverse
corpora. Despite its promise, zero-shot methods that attempt to operationalize
these outputs have met with limited success. We hypothesize that one of the
problems is that they use the mean to aggregate next-token distribution metrics
across tokens, when some tokens are naturally easier or harder to predict and
should be weighted differently. Based on this idea, we propose the Perplexity
Attention Weighted Network (PAWN), which uses the last hidden states of the LLM
and positions to weight the sum of a series of features based on metrics from
the next-token distribution across the sequence length. Although not zero-shot,
our method allows us to cache the last hidden states and next-token
distribution metrics on disk, greatly reducing the training resource
requirements. PAWN shows competitive and even better performance
in-distribution than the strongest baselines (fine-tuned LMs) with a fraction
of their trainable parameters. Our model also generalizes better to unseen
domains and source models, with smaller variability in the decision boundary
across distribution shifts. It is also more robust to adversarial attacks, and
if the backbone has multilingual capabilities, it presents decent
generalization to languages not seen during supervised training, with LLaMA3-1B
reaching a mean macro-averaged F1 score of 81.46% in cross-validation with nine
languages.


http://arxiv.org/abs/2501.02968
FlipedRAG: Black-Box Opinion Manipulation Attacks to Retrieval-Augmented Generation of Large Language Models. (82%)
Zhuo Chen; Yuyang Gong; Miaokun Chen; Haotan Liu; Qikai Cheng; Fan Zhang; Wei Lu; Xiaozhong Liu; Jiawei Liu
  Retrieval-Augmented Generation (RAG) addresses hallucination and real-time
constraints by dynamically retrieving relevant information from a knowledge
database to supplement the LLMs' input. When presented with a query, RAG
selects the most semantically similar texts from its knowledge bases and uses
them as context for the LLMs to generate more accurate responses. RAG also
creates a new attack surface, especially since RAG databases are frequently
sourced from public domains. While existing studies have predominantly focused
on optimizing RAG's performance and efficiency, emerging research has begun
addressing the security concerns associated with RAG. However, these works have
some limitations, typically focusing on either white-box methodologies or
heuristic-based black-box attacks. Furthermore, prior research has mainly
targeted simple factoid question answering, which is neither practically
challenging nor resistant to correction. In this paper, we unveil a more
realistic and threatening scenario: opinion manipulation for controversial
topics against RAG. Particularly, we propose a novel RAG black-box attack
method, termed FlipedRAG, which is transfer-based. By leveraging instruction
engineering, we obtain partial retrieval model outputs from black-box RAG
system, facilitating the training of surrogate models to enhance the
effectiveness of opinion manipulation attack. Extensive experimental results
confirms that our approach significantly enhances the average success rate of
opinion manipulation by 16.7%. It achieves an average of a 50% directional
change in the opinion polarity of RAG responses across four themes.
Additionally, it induces a 20% shift in user cognition. Furthermore, we discuss
the efficacy of potential defense mechanisms and conclude that they are
insufficient in mitigating this type of attack, highlighting the urgent need to
develop novel defensive strategies.


http://arxiv.org/abs/2501.03507
An Empirical Study of Accuracy-Robustness Tradeoff and Training Efficiency in Self-Supervised Learning. (3%)
Fatemeh Ghofrani; Pooyan Jamshidi
  Self-supervised learning (SSL) has significantly advanced image
representation learning, yet efficiency challenges persist, particularly with
adversarial training. Many SSL methods require extensive epochs to achieve
convergence, a demand further amplified in adversarial settings. To address
this inefficiency, we revisit the robust EMP-SSL framework, emphasizing the
importance of increasing the number of crops per image to accelerate learning.
Unlike traditional contrastive learning, robust EMP-SSL leverages multi-crop
sampling, integrates an invariance term and regularization, and reduces
training epochs, enhancing time efficiency. Evaluated with both standard linear
classifiers and multi-patch embedding aggregation, robust EMP-SSL provides new
insights into SSL evaluation strategies.
  Our results show that robust crop-based EMP-SSL not only accelerates
convergence but also achieves a superior balance between clean accuracy and
adversarial robustness, outperforming multi-crop embedding aggregation.
Additionally, we extend this approach with free adversarial training in
Multi-Crop SSL, introducing the Cost-Free Adversarial Multi-Crop
Self-Supervised Learning (CF-AMC-SSL) method. CF-AMC-SSL demonstrates the
effectiveness of free adversarial training in reducing training time while
simultaneously improving clean accuracy and adversarial robustness. These
findings underscore the potential of CF-AMC-SSL for practical SSL applications.
Our code is publicly available at https://github.com/softsys4ai/CF-AMC-SSL.


http://arxiv.org/abs/2501.03301
Rethinking Byzantine Robustness in Federated Recommendation from Sparse Aggregation Perspective. (2%)
Zhongjian Zhang; Mengmei Zhang; Xiao Wang; Lingjuan Lyu; Bo Yan; Junping Du; Chuan Shi
  To preserve user privacy in recommender systems, federated recommendation
(FR) based on federated learning (FL) emerges, keeping the personal data on the
local client and updating a model collaboratively. Unlike FL, FR has a unique
sparse aggregation mechanism, where the embedding of each item is updated by
only partial clients, instead of full clients in a dense aggregation of general
FL. Recently, as an essential principle of FL, model security has received
increasing attention, especially for Byzantine attacks, where malicious clients
can send arbitrary updates. The problem of exploring the Byzantine robustness
of FR is particularly critical since in the domains applying FR, e.g.,
e-commerce, malicious clients can be injected easily by registering new
accounts. However, existing Byzantine works neglect the unique sparse
aggregation of FR, making them unsuitable for our problem. Thus, we make the
first effort to investigate Byzantine attacks on FR from the perspective of
sparse aggregation, which is non-trivial: it is not clear how to define
Byzantine robustness under sparse aggregations and design Byzantine attacks
under limited knowledge/capability. In this paper, we reformulate the Byzantine
robustness under sparse aggregation by defining the aggregation for a single
item as the smallest execution unit. Then we propose a family of effective
attack strategies, named Spattack, which exploit the vulnerability in sparse
aggregation and are categorized along the adversary's knowledge and capability.
Extensive experimental results demonstrate that Spattack can effectively
prevent convergence and even break down defenses under a few malicious clients,
raising alarms for securing FR systems.


http://arxiv.org/abs/2501.02860
Seeing the Whole in the Parts in Self-Supervised Representation Learning. (1%)
Arthur Aubret; Céline Teulière; Jochen Triesch
  Recent successes in self-supervised learning (SSL) model spatial
co-occurrences of visual features either by masking portions of an image or by
aggressively cropping it. Here, we propose a new way to model spatial
co-occurrences by aligning local representations (before pooling) with a global
image representation. We present CO-SSL, a family of instance discrimination
methods and show that it outperforms previous methods on several datasets,
including ImageNet-1K where it achieves 71.5% of Top-1 accuracy with 100
pre-training epochs. CO-SSL is also more robust to noise corruption, internal
corruption, small adversarial attacks, and large training crop sizes. Our
analysis further indicates that CO-SSL learns highly redundant local
representations, which offers an explanation for its robustness. Overall, our
work suggests that aligning local and global representations may be a powerful
principle of unsupervised category learning.


http://arxiv.org/abs/2501.02450
GCP: Guarded Collaborative Perception with Spatial-Temporal Aware Malicious Agent Detection. (67%)
Yihang Tao; Senkang Hu; Yue Hu; Haonan An; Hangcheng Cao; Yuguang Fang
  Collaborative perception significantly enhances autonomous driving safety by
extending each vehicle's perception range through message sharing among
connected and autonomous vehicles. Unfortunately, it is also vulnerable to
adversarial message attacks from malicious agents, resulting in severe
performance degradation. While existing defenses employ
hypothesis-and-verification frameworks to detect malicious agents based on
single-shot outliers, they overlook temporal message correlations, which can be
circumvented by subtle yet harmful perturbations in model input and output
spaces. This paper reveals a novel blind area confusion (BAC) attack that
compromises existing single-shot outlier-based detection methods. As a
countermeasure, we propose GCP, a Guarded Collaborative Perception framework
based on spatial-temporal aware malicious agent detection, which maintains
single-shot spatial consistency through a confidence-scaled spatial concordance
loss, while simultaneously examining temporal anomalies by reconstructing
historical bird's eye view motion flows in low-confidence regions. We also
employ a joint spatial-temporal Benjamini-Hochberg test to synthesize
dual-domain anomaly results for reliable malicious agent detection. Extensive
experiments demonstrate GCP's superior performance under diverse attack
scenarios, achieving up to 34.69% improvements in AP@0.5 compared to the
state-of-the-art CP defense strategies under BAC attacks, while maintaining
consistent 5-8% improvements under other typical attacks. Code will be released
at https://github.com/CP-Security/GCP.git.


http://arxiv.org/abs/2501.02629
Layer-Level Self-Exposure and Patch: Affirmative Token Mitigation for Jailbreak Attack Defense. (62%)
Yang Ouyang; Hengrui Gu; Shuhang Lin; Wenyue Hua; Jie Peng; Bhavya Kailkhura; Tianlong Chen; Kaixiong Zhou
  As large language models (LLMs) are increasingly deployed in diverse
applications, including chatbot assistants and code generation, aligning their
behavior with safety and ethical standards has become paramount. However,
jailbreak attacks, which exploit vulnerabilities to elicit unintended or
harmful outputs, threaten LLMs' safety significantly. In this paper, we
introduce Layer-AdvPatcher, a novel methodology designed to defend against
jailbreak attacks by utilizing an unlearning strategy to patch specific layers
within LLMs through self-augmented datasets. Our insight is that certain
layer(s), tend to produce affirmative tokens when faced with harmful prompts.
By identifying these layers and adversarially exposing them to generate more
harmful data, one can understand their inherent and diverse vulnerabilities to
attacks. With these exposures, we then "unlearn" these issues, reducing the
impact of affirmative tokens and hence minimizing jailbreak risks while keeping
the model's responses to safe queries intact. We conduct extensive experiments
on two models, four benchmark datasets, and multiple state-of-the-art jailbreak
benchmarks to demonstrate the efficacy of our approach. Results indicate that
our framework reduces the harmfulness and attack success rate of jailbreak
attacks without compromising utility for benign queries compared to recent
defense methods.


http://arxiv.org/abs/2501.02654
Tougher Text, Smarter Models: Raising the Bar for Adversarial Defence Benchmarks. (54%)
Yang Wang; Chenghua Lin
  vulnerability of deep learning models to adversarial attacks. While various
defence mechanisms have been proposed, there is a lack of comprehensive
benchmarks that evaluate these defences across diverse datasets, models, and
tasks. In this work, we address this gap by presenting an extensive benchmark
for textual adversarial defence that significantly expands upon previous work.
Our benchmark incorporates a wide range of datasets, evaluates state-of-the-art
defence mechanisms, and extends the assessment to include critical tasks such
as single-sentence classification, similarity and paraphrase identification,
natural language inference, and commonsense reasoning. This work not only
serves as a valuable resource for researchers and practitioners in the field of
adversarial robustness but also identifies key areas for future research in
textual adversarial defence. By establishing a new standard for benchmarking in
this domain, we aim to accelerate progress towards more robust and reliable
natural language processing systems.


http://arxiv.org/abs/2501.02704
Persistence of Backdoor-based Watermarks for Neural Networks: A Comprehensive Evaluation. (5%)
Anh Tu Ngo; Chuan Song Heng; Nandish Chattopadhyay; Anupam Chattopadhyay
  Deep Neural Networks (DNNs) have gained considerable traction in recent years
due to the unparalleled results they gathered. However, the cost behind
training such sophisticated models is resource intensive, resulting in many to
consider DNNs to be intellectual property (IP) to model owners. In this era of
cloud computing, high-performance DNNs are often deployed all over the internet
so that people can access them publicly. As such, DNN watermarking schemes,
especially backdoor-based watermarks, have been actively developed in recent
years to preserve proprietary rights. Nonetheless, there lies much uncertainty
on the robustness of existing backdoor watermark schemes, towards both
adversarial attacks and unintended means such as fine-tuning neural network
models. One reason for this is that no complete guarantee of robustness can be
assured in the context of backdoor-based watermark. In this paper, we
extensively evaluate the persistence of recent backdoor-based watermarks within
neural networks in the scenario of fine-tuning, we propose/develop a novel
data-driven idea to restore watermark after fine-tuning without exposing the
trigger set. Our empirical results show that by solely introducing training
data after fine-tuning, the watermark can be restored if model parameters do
not shift dramatically during fine-tuning. Depending on the types of trigger
samples used, trigger accuracy can be reinstated to up to 100%. Our study
further explores how the restoration process works using loss landscape
visualization, as well as the idea of introducing training data in fine-tuning
stage to alleviate watermark vanishing.


http://arxiv.org/abs/2501.02232
Distillation-Enhanced Physical Adversarial Attacks. (96%)
Wei Liu; Yonglin Wu; Chaoqun Li; Zhuodong Liu; Huanqian Yan
  The study of physical adversarial patches is crucial for identifying
vulnerabilities in AI-based recognition systems and developing more robust deep
learning models. While recent research has focused on improving patch
stealthiness for greater practical applicability, achieving an effective
balance between stealth and attack performance remains a significant challenge.
To address this issue, we propose a novel physical adversarial attack method
that leverages knowledge distillation. Specifically, we first define a stealthy
color space tailored to the target environment to ensure smooth blending. Then,
we optimize an adversarial patch in an unconstrained color space, which serves
as the 'teacher' patch. Finally, we use an adversarial knowledge distillation
module to transfer the teacher patch's knowledge to the 'student' patch,
guiding the optimization of the stealthy patch. Experimental results show that
our approach improves attack performance by 20%, while maintaining stealth,
highlighting its practical value.


http://arxiv.org/abs/2501.03272
Backdoor Token Unlearning: Exposing and Defending Backdoors in Pretrained Language Models. (75%)
Peihai Jiang; Xixiang Lyu; Yige Li; Jing Ma
  Supervised fine-tuning has become the predominant method for adapting large
pretrained models to downstream tasks. However, recent studies have revealed
that these models are vulnerable to backdoor attacks, where even a small number
of malicious samples can successfully embed backdoor triggers into the model.
While most existing defense methods focus on post-training backdoor defense,
efficiently defending against backdoor attacks during training phase remains
largely unexplored. To address this gap, we propose a novel defense method
called Backdoor Token Unlearning (BTU), which proactively detects and
neutralizes trigger tokens during the training stage. Our work is based on two
key findings: 1) backdoor learning causes distinctive differences between
backdoor token parameters and clean token parameters in word embedding layers,
and 2) the success of backdoor attacks heavily depends on backdoor token
parameters. The BTU defense leverages these properties to identify aberrant
embedding parameters and subsequently removes backdoor behaviors using a
fine-grained unlearning technique. Extensive evaluations across three datasets
and four types of backdoor attacks demonstrate that BTU effectively defends
against these threats while preserving the model's performance on primary
tasks. Our code is available at https://github.com/XDJPH/BTU.


http://arxiv.org/abs/2501.02373
BADTV: Unveiling Backdoor Threats in Third-Party Task Vectors. (22%)
Chia-Yi Hsu; Yu-Lin Tsai; Yu Zhe; Yan-Lun Chen; Chih-Hsun Lin; Chia-Mu Yu; Yang Zhang; Chun-Ying Huang; Jun Sakuma
  Task arithmetic in large-scale pre-trained models enables flexible adaptation
to diverse downstream tasks without extensive re-training. By leveraging task
vectors (TVs), users can perform modular updates to pre-trained models through
simple arithmetic operations like addition and subtraction. However, this
flexibility introduces new security vulnerabilities. In this paper, we identify
and evaluate the susceptibility of TVs to backdoor attacks, demonstrating how
malicious actors can exploit TVs to compromise model integrity. By developing
composite backdoors and eliminating redudant clean tasks, we introduce BadTV, a
novel backdoor attack specifically designed to remain effective under task
learning, forgetting, and analogies operations. Our extensive experiments
reveal that BadTV achieves near-perfect attack success rates across various
scenarios, significantly impacting the security of models using task
arithmetic. We also explore existing defenses, showing that current methods
fail to detect or mitigate BadTV. Our findings highlight the need for robust
defense mechanisms to secure TVs in real-world applications, especially as TV
services become more popular in machine-learning ecosystems.


http://arxiv.org/abs/2501.02042
Towards Robust and Accurate Stability Estimation of Local Surrogate Models in Text-based Explainable AI. (98%)
Christopher Burger; Charles Walter; Thai Le; Lingwei Chen
  Recent work has investigated the concept of adversarial attacks on
explainable AI (XAI) in the NLP domain with a focus on examining the
vulnerability of local surrogate methods such as Lime to adversarial
perturbations or small changes on the input of a machine learning (ML) model.
In such attacks, the generated explanation is manipulated while the meaning and
structure of the original input remain similar under the ML model. Such attacks
are especially alarming when XAI is used as a basis for decision making (e.g.,
prescribing drugs based on AI medical predictors) or for legal action (e.g.,
legal dispute involving AI software). Although weaknesses across many XAI
methods have been shown to exist, the reasons behind why remain little
explored. Central to this XAI manipulation is the similarity measure used to
calculate how one explanation differs from another. A poor choice of similarity
measure can lead to erroneous conclusions about the stability or adversarial
robustness of an XAI method. Therefore, this work investigates a variety of
similarity measures designed for text-based ranked lists referenced in related
work to determine their comparative suitability for use. We find that many
measures are overly sensitive, resulting in erroneous estimates of stability.
We then propose a weighting scheme for text-based data that incorporates the
synonymity between the features within an explanation, providing more accurate
estimates of the actual weakness of XAI methods to adversarial examples.


http://arxiv.org/abs/2501.02147
Exploring Secure Machine Learning Through Payload Injection and FGSM Attacks on ResNet-50. (93%)
Umesh Yadav; Suman Niraula; Gaurav Kumar Gupta; Bicky Yadav
  This paper investigates the resilience of a ResNet-50 image classification
model under two prominent security threats: Fast Gradient Sign Method (FGSM)
adversarial attacks and malicious payload injection. Initially, the model
attains a 53.33% accuracy on clean images. When subjected to FGSM
perturbations, its overall accuracy remains unchanged; however, the model's
confidence in incorrect predictions notably increases. Concurrently, a payload
injection scheme is successfully executed in 93.33% of the tested samples,
revealing how stealthy attacks can manipulate model predictions without
degrading visual quality. These findings underscore the vulnerability of even
high-performing neural networks and highlight the urgency of developing more
robust defense mechanisms for security-critical applications.


http://arxiv.org/abs/2501.01908
Detecting and Mitigating Adversarial Attacks on Deep Learning-Based MRI Reconstruction Without Any Retraining. (84%)
Mahdi Saberi; Chi Zhang; Mehmet Akcakaya
  Deep learning (DL) methods, especially those based on physics-driven DL, have
become the state-of-the-art for reconstructing sub-sampled magnetic resonance
imaging (MRI) data. However, studies have shown that these methods are
susceptible to small adversarial input perturbations, or attacks, resulting in
major distortions in the output images. Various strategies have been proposed
to reduce the effects of these attacks, but they require retraining and may
lower reconstruction quality for non-perturbed/clean inputs. In this work, we
propose a novel approach for detecting and mitigating adversarial attacks on
MRI reconstruction models without any retraining. Our detection strategy is
based on the idea of cyclic measurement consistency. The output of the model is
mapped to another set of MRI measurements for a different sub-sampling pattern,
and this synthesized data is reconstructed with the same model. Intuitively,
without an attack, the second reconstruction is expected to be consistent with
the first, while with an attack, disruptions are present. Subsequently, this
idea is extended to devise a novel objective function, which is minimized
within a small ball around the attack input for mitigation. Experimental
results show that our method substantially reduces the impact of adversarial
perturbations across different datasets, attack types/strengths and PD-DL
networks, and qualitatively and quantitatively outperforms conventional
mitigation methods that involve retraining.


http://arxiv.org/abs/2501.01818
Rerouting LLM Routers. (82%)
Avital Shafran; Roei Schuster; Thomas Ristenpart; Vitaly Shmatikov
  LLM routers aim to balance quality and cost of generation by classifying
queries and routing them to a cheaper or more expensive LLM depending on their
complexity. Routers represent one type of what we call LLM control planes:
systems that orchestrate use of one or more LLMs. In this paper, we investigate
routers' adversarial robustness.
  We first define LLM control plane integrity, i.e., robustness of LLM
orchestration to adversarial inputs, as a distinct problem in AI safety. Next,
we demonstrate that an adversary can generate query-independent token sequences
we call ``confounder gadgets'' that, when added to any query, cause LLM routers
to send the query to a strong LLM.
  Our quantitative evaluation shows that this attack is successful both in
white-box and black-box settings against a variety of open-source and
commercial routers, and that confounding queries do not affect the quality of
LLM responses. Finally, we demonstrate that gadgets can be effective while
maintaining low perplexity, thus perplexity-based filtering is not an effective
defense. We finish by investigating alternative defenses.


http://arxiv.org/abs/2501.02135
AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMs. (69%)
Sanjoy Chowdhury; Sayan Nag; Subhrajyoti Dasgupta; Yaoting Wang; Mohamed Elhoseiny; Ruohan Gao; Dinesh Manocha
  With the rapid advancement of Multi-modal Large Language Models (MLLMs),
several diagnostic benchmarks have recently been developed to assess these
models' multi-modal reasoning proficiency. However, these benchmarks are
restricted to assessing primarily the visual aspect and do not examine the
holistic audio-visual (AV) understanding. Moreover, currently, there are no
benchmarks that investigate the capabilities of AVLLMs to calibrate their
responses when presented with perturbed inputs. To this end, we introduce
Audio-Visual Trustworthiness assessment Benchmark (AVTrustBench), comprising
600K samples spanning over 9 meticulously crafted tasks, evaluating the
capabilities of AVLLMs across three distinct dimensions: Adversarial attack,
Compositional reasoning, and Modality-specific dependency. Using our benchmark
we extensively evaluate 13 state-of-the-art AVLLMs. The findings reveal that
the majority of existing models fall significantly short of achieving
human-like comprehension, offering valuable insights for future research
directions. To alleviate the limitations in the existing approaches, we further
propose a robust, model-agnostic calibrated audio-visual preference
optimization based training strategy CAVPref, obtaining a gain up to 30.19%
across all 9 tasks. We will publicly release our code and benchmark to
facilitate future research in this direction.


http://arxiv.org/abs/2501.01872
Turning Logic Against Itself : Probing Model Defenses Through Contrastive Questions. (62%)
Rachneet Sachdeva; Rima Hazra; Iryna Gurevych
  Despite significant efforts to align large language models with human values
and ethical guidelines, these models remain susceptible to sophisticated
jailbreak attacks that exploit their reasoning capabilities. Traditional safety
mechanisms often focus on detecting explicit malicious intent, leaving deeper
vulnerabilities unaddressed. In this work, we introduce a jailbreak technique,
POATE (Polar Opposite query generation, Adversarial Template construction, and
Elaboration), which leverages contrastive reasoning to elicit unethical
responses. POATE generates prompts with semantically opposite intents and
combines them with adversarial templates to subtly direct models toward
producing harmful responses. We conduct extensive evaluations across six
diverse language model families of varying parameter sizes, including LLaMA3,
Gemma2, Phi3, and GPT-4, to demonstrate the robustness of the attack, achieving
significantly higher attack success rates (~44%) compared to existing methods.
We evaluate our proposed attack against seven safety defenses, revealing their
limitations in addressing reasoning-based vulnerabilities. To counteract this,
we propose a defense strategy that improves reasoning robustness through
chain-of-thought prompting and reverse thinking, mitigating reasoning-driven
adversarial exploits.


http://arxiv.org/abs/2501.01913
Mingling with the Good to Backdoor Federated Learning. (12%)
Nuno Neves
  Federated learning (FL) is a decentralized machine learning technique that
allows multiple entities to jointly train a model while preserving dataset
privacy. However, its distributed nature has raised various security concerns,
which have been addressed by increasingly sophisticated defenses. These
protections utilize a range of data sources and metrics to, for example, filter
out malicious model updates, ensuring that the impact of attacks is minimized
or eliminated.
  This paper explores the feasibility of designing a generic attack method
capable of installing backdoors in FL while evading a diverse array of
defenses. Specifically, we focus on an attacker strategy called MIGO, which
aims to produce model updates that subtly blend with legitimate ones. The
resulting effect is a gradual integration of a backdoor into the global model,
often ensuring its persistence long after the attack concludes, while
generating enough ambiguity to hinder the effectiveness of defenses.
  MIGO was employed to implant three types of backdoors across five datasets
and different model architectures. The results demonstrate the significant
threat posed by these backdoors, as MIGO consistently achieved exceptionally
high backdoor accuracy (exceeding 90%) while maintaining the utility of the
main task. Moreover, MIGO exhibited strong evasion capabilities against ten
defenses, including several state-of-the-art methods. When compared to four
other attack strategies, MIGO consistently outperformed them across most
configurations. Notably, even in extreme scenarios where the attacker controls
just 0.1% of the clients, the results indicate that successful backdoor
insertion is possible if the attacker can persist for a sufficient number of
rounds.


http://arxiv.org/abs/2501.02182
AdaMixup: A Dynamic Defense Framework for Membership Inference Attack Mitigation. (10%)
Ying Chen; Jiajing Chen; Yijie Weng; ChiaHua Chang; Dezhi Yu; Guanbiao Lin
  Membership inference attacks have emerged as a significant privacy concern in
the training of deep learning models, where attackers can infer whether a data
point was part of the training set based on the model's outputs. To address
this challenge, we propose a novel defense mechanism, AdaMixup. AdaMixup
employs adaptive mixup techniques to enhance the model's robustness against
membership inference attacks by dynamically adjusting the mixup strategy during
training. This method not only improves the model's privacy protection but also
maintains high performance. Experimental results across multiple datasets
demonstrate that AdaMixup significantly reduces the risk of membership
inference attacks while achieving a favorable trade-off between defensive
efficiency and model accuracy. This research provides an effective solution for
data privacy protection and lays the groundwork for future advancements in
mixup training methods.


http://arxiv.org/abs/2501.01741
How Toxic Can You Get? Search-based Toxicity Testing for Large Language Models. (9%)
Simone Corbo; Luca Bancale; Gennaro Valeria De; Livia Lestingi; Vincenzo Scotti; Matteo Camilli
  Language is a deep-rooted means of perpetration of stereotypes and
discrimination. Large Language Models (LLMs), now a pervasive technology in our
everyday lives, can cause extensive harm when prone to generating toxic
responses. The standard way to address this issue is to align the LLM, which,
however, dampens the issue without constituting a definitive solution.
Therefore, testing LLM even after alignment efforts remains crucial for
detecting any residual deviations with respect to ethical standards. We present
EvoTox, an automated testing framework for LLMs' inclination to toxicity,
providing a way to quantitatively assess how much LLMs can be pushed towards
toxic responses even in the presence of alignment. The framework adopts an
iterative evolution strategy that exploits the interplay between two LLMs, the
System Under Test (SUT) and the Prompt Generator steering SUT responses toward
higher toxicity. The toxicity level is assessed by an automated oracle based on
an existing toxicity classifier. We conduct a quantitative and qualitative
empirical evaluation using four state-of-the-art LLMs as evaluation subjects
having increasing complexity (7-13 billion parameters). Our quantitative
evaluation assesses the cost-effectiveness of four alternative versions of
EvoTox against existing baseline methods, based on random search, curated
datasets of toxic prompts, and adversarial attacks. Our qualitative assessment
engages human evaluators to rate the fluency of the generated prompts and the
perceived toxicity of the responses collected during the testing sessions.
Results indicate that the effectiveness, in terms of detected toxicity level,
is significantly higher than the selected baseline methods (effect size up to
1.0 against random search and up to 0.99 against adversarial attacks).
Furthermore, EvoTox yields a limited cost overhead (from 22% to 35% on
average).


http://arxiv.org/abs/2501.01620
Adaptive Meta-learning-based Adversarial Training for Robust Automatic Modulation Classification. (99%)
Amirmohammad Bamdad; Ali Owfi; Fatemeh Afghah
  DL-based automatic modulation classification (AMC) models are highly
susceptible to adversarial attacks, where even minimal input perturbations can
cause severe misclassifications. While adversarially training an AMC model
based on an adversarial attack significantly increases its robustness against
that attack, the AMC model will still be defenseless against other adversarial
attacks. The theoretically infinite possibilities for adversarial perturbations
mean that an AMC model will inevitably encounter new unseen adversarial attacks
if it is ever to be deployed to a real-world communication system. Moreover,
the computational limitations and challenges of obtaining new data in real-time
will not allow a full training process for the AMC model to adapt to the new
attack when it is online. To this end, we propose a meta-learning-based
adversarial training framework for AMC models that substantially enhances
robustness against unseen adversarial attacks and enables fast adaptation to
these attacks using just a few new training samples, if any are available. Our
results demonstrate that this training framework provides superior robustness
and accuracy with much less online training time than conventional adversarial
training of AMC models, making it highly efficient for real-world deployment.


http://arxiv.org/abs/2501.01106
AIM: Additional Image Guided Generation of Transferable Adversarial Attacks. (99%)
Teng Li; Xingjun Ma; Yu-Gang Jiang
  Transferable adversarial examples highlight the vulnerability of deep neural
networks (DNNs) to imperceptible perturbations across various real-world
applications. While there have been notable advancements in untargeted
transferable attacks, targeted transferable attacks remain a significant
challenge. In this work, we focus on generative approaches for targeted
transferable attacks. Current generative attacks focus on reducing overfitting
to surrogate models and the source data domain, but they often overlook the
importance of enhancing transferability through additional semantics. To
address this issue, we introduce a novel plug-and-play module into the general
generator architecture to enhance adversarial transferability. Specifically, we
propose a \emph{Semantic Injection Module} (SIM) that utilizes the semantics
contained in an additional guiding image to improve transferability. The
guiding image provides a simple yet effective method to incorporate target
semantics from the target class to create targeted and highly transferable
attacks. Additionally, we propose new loss formulations that can integrate the
semantic injection module more effectively for both targeted and untargeted
attacks. We conduct comprehensive experiments under both targeted and
untargeted attack settings to demonstrate the efficacy of our proposed
approach.


http://arxiv.org/abs/2501.01090
HoneypotNet: Backdoor Attacks Against Model Extraction. (93%)
Yixu Wang; Tianle Gu; Yan Teng; Yingchun Wang; Xingjun Ma
  Model extraction attacks are one type of inference-time attacks that
approximate the functionality and performance of a black-box victim model by
launching a certain number of queries to the model and then leveraging the
model's predictions to train a substitute model. These attacks pose severe
security threats to production models and MLaaS platforms and could cause
significant monetary losses to the model owners. A body of work has proposed to
defend machine learning models against model extraction attacks, including both
active defense methods that modify the model's outputs or increase the query
overhead to avoid extraction and passive defense methods that detect malicious
queries or leverage watermarks to perform post-verification. In this work, we
introduce a new defense paradigm called attack as defense which modifies the
model's output to be poisonous such that any malicious users that attempt to
use the output to train a substitute model will be poisoned. To this end, we
propose a novel lightweight backdoor attack method dubbed HoneypotNet that
replaces the classification layer of the victim model with a honeypot layer and
then fine-tunes the honeypot layer with a shadow model (to simulate model
extraction) via bi-level optimization to modify its output to be poisonous
while remaining the original performance. We empirically demonstrate on four
commonly used benchmark datasets that HoneypotNet can inject backdoors into
substitute models with a high success rate. The injected backdoor not only
facilitates ownership verification but also disrupts the functionality of
substitute models, serving as a significant deterrent to model extraction
attacks.


http://arxiv.org/abs/2501.01516
Improving Robustness Estimates in Natural Language Explainable AI though Synonymity Weighted Similarity Measures. (87%)
Christopher Burger
  Explainable AI (XAI) has seen a surge in recent interest with the
proliferation of powerful but intractable black-box models. Moreover, XAI has
come under fire for techniques that may not offer reliable explanations. As
many of the methods in XAI are themselves models, adversarial examples have
been prominent in the literature surrounding the effectiveness of XAI, with the
objective of these examples being to alter the explanation while maintaining
the output of the original model. For explanations in natural language, it is
natural to use measures found in the domain of information retrieval for use
with ranked lists to guide the adversarial XAI process. We show that the
standard implementation of these measures are poorly suited for the comparison
of explanations in adversarial XAI and amend them by using information that is
discarded, the synonymity of perturbed words. This synonymity weighting
produces more accurate estimates of the actual weakness of XAI methods to
adversarial examples.


http://arxiv.org/abs/2501.01263
Stealthy Backdoor Attack to Real-world Models in Android Apps. (80%)
Jiali Wei; Ming Fan; Xicheng Zhang; Wenjing Jiao; Haijun Wang; Ting Liu
  Powered by their superior performance, deep neural networks (DNNs) have found
widespread applications across various domains. Many deep learning (DL) models
are now embedded in mobile apps, making them more accessible to end users
through on-device DL. However, deploying on-device DL to users' smartphones
simultaneously introduces several security threats. One primary threat is
backdoor attacks. Extensive research has explored backdoor attacks for several
years and has proposed numerous attack approaches. However, few studies have
investigated backdoor attacks on DL models deployed in the real world, or they
have shown obvious deficiencies in effectiveness and stealthiness. In this
work, we explore more effective and stealthy backdoor attacks on real-world DL
models extracted from mobile apps. Our main justification is that imperceptible
and sample-specific backdoor triggers generated by DNN-based steganography can
enhance the efficacy of backdoor attacks on real-world models. We first confirm
the effectiveness of steganography-based backdoor attacks on four
state-of-the-art DNN models. Subsequently, we systematically evaluate and
analyze the stealthiness of the attacks to ensure they are difficult to
perceive. Finally, we implement the backdoor attacks on real-world models and
compare our approach with three baseline methods. We collect 38,387 mobile
apps, extract 89 DL models from them, and analyze these models to obtain the
prerequisite model information for the attacks. After identifying the target
models, our approach achieves an average of 12.50% higher attack success rate
than DeepPayload while better maintaining the normal performance of the models.
Extensive experimental results demonstrate that our method enables more
effective, robust, and stealthy backdoor attacks on real-world models.


http://arxiv.org/abs/2501.01529
SAFER: Sharpness Aware layer-selective Finetuning for Enhanced Robustness in vision transformers. (45%)
Bhavna Gopal; Huanrui Yang; Mark Horton; Yiran Chen
  Vision transformers (ViTs) have become essential backbones in advanced
computer vision applications and multi-modal foundation models. Despite their
strengths, ViTs remain vulnerable to adversarial perturbations, comparable to
or even exceeding the vulnerability of convolutional neural networks (CNNs).
Furthermore, the large parameter count and complex architecture of ViTs make
them particularly prone to adversarial overfitting, often compromising both
clean and adversarial accuracy.
  This paper mitigates adversarial overfitting in ViTs through a novel,
layer-selective fine-tuning approach: SAFER. Instead of optimizing the entire
model, we identify and selectively fine-tune a small subset of layers most
susceptible to overfitting, applying sharpness-aware minimization to these
layers while freezing the rest of the model. Our method consistently enhances
both clean and adversarial accuracy over baseline approaches. Typical
improvements are around 5%, with some cases achieving gains as high as 20%
across various ViT architectures and datasets.


http://arxiv.org/abs/2501.01606
Test Input Validation for Vision-based DL Systems: An Active Learning Approach. (1%)
Delaram Ghobari; Mohammad Hossein Amini; Dai Quoc Tran; Seunghee Park; Shiva Nejati; Mehrdad Sabetzadeh
  Testing deep learning (DL) systems requires extensive and diverse, yet valid,
test inputs. While synthetic test input generation methods, such as metamorphic
testing, are widely used for DL testing, they risk introducing invalid inputs
that do not accurately reflect real-world scenarios. Invalid test inputs can
lead to misleading results. Hence, there is a need for automated validation of
test inputs to ensure effective assessment of DL systems. In this paper, we
propose a test input validation approach for vision-based DL systems. Our
approach uses active learning to balance the trade-off between accuracy and the
manual effort required for test input validation. Further, by employing
multiple image-comparison metrics, it achieves better results in classifying
valid and invalid test inputs compared to methods that rely on single metrics.
We evaluate our approach using an industrial and a public-domain dataset. Our
evaluation shows that our multi-metric, active learning-based approach produces
several optimal accuracy-effort trade-offs, including those deemed practical
and desirable by our industry partner. Furthermore, provided with the same
level of manual effort, our approach is significantly more accurate than two
state-of-the-art test input validation methods, achieving an average accuracy
of 97%. Specifically, the use of multiple metrics, rather than a single metric,
results in an average improvement of at least 5.4% in overall accuracy compared
to the state-of-the-art baselines. Incorporating an active learning loop for
test input validation yields an additional 7.5% improvement in average
accuracy, bringing the overall average improvement of our approach to at least
12.9% compared to the baselines.


http://arxiv.org/abs/2501.01558
Predicting the Performance of Black-box LLMs through Self-Queries. (1%)
Dylan Sam; Marc Finzi; J. Zico Kolter
  As large language models (LLMs) are increasingly relied on in AI systems,
predicting when they make mistakes is crucial. While a great deal of work in
the field uses internal representations to interpret model behavior, these
representations are inaccessible when given solely black-box access through an
API. In this paper, we extract features of LLMs in a black-box manner by using
follow-up prompts and taking the probabilities of different responses as
representations to train reliable predictors of model behavior. We demonstrate
that training a linear model on these low-dimensional representations produces
reliable and generalizable predictors of model performance at the instance
level (e.g., if a particular generation correctly answers a question).
Remarkably, these can often outperform white-box linear predictors that operate
over a model's hidden state or the full distribution over its vocabulary. In
addition, we demonstrate that these extracted features can be used to evaluate
more nuanced aspects of a language model's state. For instance, they can be
used to distinguish between a clean version of GPT-4o-mini and a version that
has been influenced via an adversarial system prompt that answers
question-answering tasks incorrectly or introduces bugs into generated code.
Furthermore, they can reliably distinguish between different model
architectures and sizes, enabling the detection of misrepresented models
provided through an API (e.g., identifying if GPT-3.5 is supplied instead of
GPT-4o-mini).


http://arxiv.org/abs/2501.01015
Boosting Adversarial Transferability with Spatial Adversarial Alignment. (99%)
Zhaoyu Chen; Haijing Guo; Kaixun Jiang; Jiyuan Fu; Xinyu Zhou; Dingkang Yang; Hao Tang; Bo Li; Wenqiang Zhang
  Deep neural networks are vulnerable to adversarial examples that exhibit
transferability across various models. Numerous approaches are proposed to
enhance the transferability of adversarial examples, including advanced
optimization, data augmentation, and model modifications. However, these
methods still show limited transferability, particularly in cross-architecture
scenarios, such as from CNN to ViT. To achieve high transferability, we propose
a technique termed Spatial Adversarial Alignment (SAA), which employs an
alignment loss and leverages a witness model to fine-tune the surrogate model.
Specifically, SAA consists of two key parts: spatial-aware alignment and
adversarial-aware alignment. First, we minimize the divergences of features
between the two models in both global and local regions, facilitating spatial
alignment. Second, we introduce a self-adversarial strategy that leverages
adversarial examples to impose further constraints, aligning features from an
adversarial perspective. Through this alignment, the surrogate model is trained
to concentrate on the common features extracted by the witness model. This
facilitates adversarial attacks on these shared features, thereby yielding
perturbations that exhibit enhanced transferability. Extensive experiments on
various architectures on ImageNet show that aligned surrogate models based on
SAA can provide higher transferable adversarial examples, especially in
cross-architecture attacks.


http://arxiv.org/abs/2501.01025
Towards Adversarially Robust Deep Metric Learning. (99%)
Xiaopeng Ke
  Deep Metric Learning (DML) has shown remarkable successes in many domains by
taking advantage of powerful deep neural networks. Deep neural networks are
prone to adversarial attacks and could be easily fooled by adversarial
examples. The current progress on this robustness issue is mainly about deep
classification models but pays little attention to DML models. Existing works
fail to thoroughly inspect the robustness of DML and neglect an important DML
scenario, the clustering-based inference. In this work, we first point out the
robustness issue of DML models in clustering-based inference scenarios. We find
that, for the clustering-based inference, existing defenses designed DML are
unable to be reused and the adaptions of defenses designed for deep
classification models cannot achieve satisfactory robustness performance. To
alleviate the hazard of adversarial examples, we propose a new defense, the
Ensemble Adversarial Training (EAT), which exploits ensemble learning and
adversarial training. EAT promotes the diversity of the ensemble, encouraging
each model in the ensemble to have different robustness features, and employs a
self-transferring mechanism to make full use of the robustness statistics of
the whole ensemble in the update of every single model. We evaluate the EAT
method on three widely-used datasets with two popular model architectures. The
results show that the proposed EAT method greatly outperforms the adaptions of
defenses designed for deep classification models.


http://arxiv.org/abs/2501.01042
Image-based Multimodal Models as Intruders: Transferable Multimodal Attacks on Video-based MLLMs. (99%)
Linhao Huang; Xue Jiang; Zhiqiang Wang; Wentao Mo; Xi Xiao; Bo Han; Yongjie Yin; Feng Zheng
  Video-based multimodal large language models (V-MLLMs) have shown
vulnerability to adversarial examples in video-text multimodal tasks. However,
the transferability of adversarial videos to unseen models--a common and
practical real world scenario--remains unexplored. In this paper, we pioneer an
investigation into the transferability of adversarial video samples across
V-MLLMs. We find that existing adversarial attack methods face significant
limitations when applied in black-box settings for V-MLLMs, which we attribute
to the following shortcomings: (1) lacking generalization in perturbing video
features, (2) focusing only on sparse key-frames, and (3) failing to integrate
multimodal information. To address these limitations and deepen the
understanding of V-MLLM vulnerabilities in black-box scenarios, we introduce
the Image-to-Video MLLM (I2V-MLLM) attack. In I2V-MLLM, we utilize an
image-based multimodal model (IMM) as a surrogate model to craft adversarial
video samples. Multimodal interactions and temporal information are integrated
to disrupt video representations within the latent space, improving adversarial
transferability. In addition, a perturbation propagation technique is
introduced to handle different unknown frame sampling strategies. Experimental
results demonstrate that our method can generate adversarial examples that
exhibit strong transferability across different V-MLLMs on multiple video-text
multimodal tasks. Compared to white-box attacks on these models, our black-box
attacks (using BLIP-2 as surrogate model) achieve competitive performance, with
average attack success rates of 55.48% on MSVD-QA and 58.26% on MSRVTT-QA for
VideoQA tasks, respectively. Our code will be released upon acceptance.


http://arxiv.org/abs/2501.00745
Dynamics of Adversarial Attacks on Large Language Model-Based Search Engines. (76%)
Xiyang Hu
  The increasing integration of Large Language Model (LLM) based search engines
has transformed the landscape of information retrieval. However, these systems
are vulnerable to adversarial attacks, especially ranking manipulation attacks,
where attackers craft webpage content to manipulate the LLM's ranking and
promote specific content, gaining an unfair advantage over competitors. In this
paper, we study the dynamics of ranking manipulation attacks. We frame this
problem as an Infinitely Repeated Prisoners' Dilemma, where multiple players
strategically decide whether to cooperate or attack. We analyze the conditions
under which cooperation can be sustained, identifying key factors such as
attack costs, discount rates, attack success rates, and trigger strategies that
influence player behavior. We identify tipping points in the system dynamics,
demonstrating that cooperation is more likely to be sustained when players are
forward-looking. However, from a defense perspective, we find that simply
reducing attack success probabilities can, paradoxically, incentivize attacks
under certain conditions. Furthermore, defensive measures to cap the upper
bound of attack success rates may prove futile in some scenarios. These
insights highlight the complexity of securing LLM-based systems. Our work
provides a theoretical foundation and practical insights for understanding and
mitigating their vulnerabilities, while emphasizing the importance of adaptive
security strategies and thoughtful ecosystem design.


http://arxiv.org/abs/2501.00879
TrustRAG: Enhancing Robustness and Trustworthiness in RAG. (16%)
Huichi Zhou; Kin-Hei Lee; Zhonghao Zhan; Yue Chen; Zhenhao Li
  Retrieval-Augmented Generation (RAG) systems enhance large language models
(LLMs) by integrating external knowledge sources, enabling more accurate and
contextually relevant responses tailored to user queries. However, these
systems remain vulnerable to corpus poisoning attacks that can significantly
degrade LLM performance through the injection of malicious content. To address
these challenges, we propose TrustRAG, a robust framework that systematically
filters compromised and irrelevant content before it reaches the language
model. Our approach implements a two-stage defense mechanism: first, it employs
K-means clustering to identify potential attack patterns in retrieved documents
based on their semantic embeddings, effectively isolating suspicious content.
Second, it leverages cosine similarity and ROUGE metrics to detect malicious
documents while resolving discrepancies between the model's internal knowledge
and external information through a self-assessment process. TrustRAG functions
as a plug-and-play, training-free module that integrates seamlessly with any
language model, whether open or closed-source, maintaining high contextual
relevance while strengthening defenses against attacks. Through extensive
experimental validation, we demonstrate that TrustRAG delivers substantial
improvements in retrieval accuracy, efficiency, and attack resistance compared
to existing approaches across multiple model architectures and datasets. We
have made TrustRAG available as open-source software at
\url{https://github.com/HuichiZhou/TrustRAG}.


http://arxiv.org/abs/2501.00824
Information Sifting Funnel: Privacy-preserving Collaborative Inference Against Model Inversion Attacks. (2%)
Rongke Liu
  The complexity of neural networks and inference tasks, coupled with demands
for computational efficiency and real-time feedback, poses significant
challenges for resource-constrained edge devices. Collaborative inference
mitigates this by assigning shallow feature extraction to edge devices and
offloading features to the cloud for further inference, reducing computational
load. However, transmitted features remain susceptible to model inversion
attacks (MIAs), which can reconstruct original input data. Current defenses,
such as perturbation and information bottleneck techniques, offer explainable
protection but face limitations, including the lack of standardized criteria
for assessing MIA difficulty, challenges in mutual information estimation, and
trade-offs among usability, privacy, and deployability.
  To address these challenges, we introduce the first criterion to evaluate MIA
difficulty in collaborative inference, supported by theoretical analysis of
existing attacks and defenses, validated using experiments with the Mutual
Information Neural Estimator (MINE). Based on these findings, we propose
SiftFunnel, a privacy-preserving framework for collaborative inference. The
edge model is trained with linear and non-linear correlation constraints to
reduce redundant information in transmitted features, enhancing privacy
protection. Label smoothing and a cloud-based upsampling module are added to
balance usability and privacy. To improve deployability, the edge model
incorporates a funnel-shaped structure and attention mechanisms, preserving
both privacy and usability. Extensive experiments demonstrate that SiftFunnel
outperforms state-of-the-art defenses against MIAs, achieving superior privacy
protection with less than 3% accuracy loss and striking an optimal balance
among usability, privacy, and practicality.


http://arxiv.org/abs/2501.00973
Defense Strategies for Autonomous Multi-agent Systems: Ensuring Safety and Resilience Under Exponentially Unbounded FDI Attacks. (1%)
Yichao Wang; Mohamadamin Rajabinezhad; Rui Liu; Shan Zuo
  False data injection (FDI) attacks pose a significant threat to autonomous
multi-agent systems (MASs). While resilient control strategies address FDI
attacks, they typically have strict assumptions on the attack signals and
overlook safety constraints, such as collision avoidance. In practical
applications, leader agents equipped with advanced sensors or weaponry span a
safe region to guide heterogeneous follower agents, ensuring coordinated
operations while addressing collision avoidance to prevent financial losses and
mission failures. This letter addresses these gaps by introducing and studying
the safety-aware and attack-resilient (SAAR) control problem under
exponentially unbounded FDI (EU-FDI) attacks. Specifically, a novel
attack-resilient observer layer (OL) is first designed to defend against EU-FDI
attacks on the OL. Then, by solving an optimization problem using the quadratic
programming (QP), the safety constraints for collision avoidance are further
integrated into the SAAR controller design to prevent collisions among
followers. An attack-resilient compensational signal is finally designed to
mitigate the adverse effects caused by the EU-FDI attack on control input layer
(CIL). Rigorous Lyapunov-based stability analysis certifies the SAAR
controller's effectiveness in ensuring both safety and resilience. This study
also pioneers a three-dimensional simulation of the SAAR containment control
problem for autonomous MASs, demonstrating its applicability in realistic
multi-agent scenarios.


http://arxiv.org/abs/2501.00707
Everywhere Attack: Attacking Locally and Globally to Boost Targeted Transferability. (99%)
Hui Zeng; Sanshuai Cui; Biwei Chen; Anjie Peng
  Adversarial examples' (AE) transferability refers to the phenomenon that AEs
crafted with one surrogate model can also fool other models. Notwithstanding
remarkable progress in untargeted transferability, its targeted counterpart
remains challenging. This paper proposes an everywhere scheme to boost targeted
transferability. Our idea is to attack a victim image both globally and
locally. We aim to optimize 'an army of targets' in every local image region
instead of the previous works that optimize a high-confidence target in the
image. Specifically, we split a victim image into non-overlap blocks and
jointly mount a targeted attack on each block. Such a strategy mitigates
transfer failures caused by attention inconsistency between surrogate and
victim models and thus results in stronger transferability. Our approach is
method-agnostic, which means it can be easily combined with existing
transferable attacks for even higher transferability. Extensive experiments on
ImageNet demonstrate that the proposed approach universally improves the
state-of-the-art targeted attacks by a clear margin, e.g., the transferability
of the widely adopted Logit attack can be improved by 28.8%-300%.We also
evaluate the crafted AEs on a real-world platform: Google Cloud Vision. Results
further support the superiority of the proposed method.


http://arxiv.org/abs/2501.00537
Extending XReason: Formal Explanations for Adversarial Detection. (82%)
Amira Jemaa; Adnan Rashid; Sofiene Tahar
  Explainable Artificial Intelligence (XAI) plays an important role in
improving the transparency and reliability of complex machine learning models,
especially in critical domains such as cybersecurity. Despite the prevalence of
heuristic interpretation methods such as SHAP and LIME, these techniques often
lack formal guarantees and may produce inconsistent local explanations. To
fulfill this need, few tools have emerged that use formal methods to provide
formal explanations. Among these, XReason uses a SAT solver to generate formal
instance-level explanation for XGBoost models. In this paper, we extend the
XReason tool to support LightGBM models as well as class-level explanations.
Additionally, we implement a mechanism to generate and detect adversarial
examples in XReason. We evaluate the efficiency and accuracy of our approach on
the CICIDS-2017 dataset, a widely used benchmark for detecting network attacks.


http://arxiv.org/abs/2412.20987
RobustBlack: Challenging Black-Box Adversarial Attacks on State-of-the-Art Defenses. (99%)
Mohamed Djilani; Salah Ghamizi; Maxime Cordy
  Although adversarial robustness has been extensively studied in white-box
settings, recent advances in black-box attacks (including transfer- and
query-based approaches) are primarily benchmarked against weak defenses,
leaving a significant gap in the evaluation of their effectiveness against more
recent and moderate robust models (e.g., those featured in the Robustbench
leaderboard). In this paper, we question this lack of attention from black-box
attacks to robust models. We establish a framework to evaluate the
effectiveness of recent black-box attacks against both top-performing and
standard defense mechanisms, on the ImageNet dataset. Our empirical evaluation
reveals the following key findings: (1) the most advanced black-box attacks
struggle to succeed even against simple adversarially trained models; (2)
robust models that are optimized to withstand strong white-box attacks, such as
AutoAttack, also exhibits enhanced resilience against black-box attacks; and
(3) robustness alignment between the surrogate models and the target model
plays a key factor in the success rate of transfer-based attacks


http://arxiv.org/abs/2412.21164
Adversarial Attack and Defense for LoRa Device Identification and Authentication via Deep Learning. (99%)
Yalin E. Sagduyu; Tugba Erpek
  LoRa provides long-range, energy-efficient communications in Internet of
Things (IoT) applications that rely on Low-Power Wide-Area Network (LPWAN)
capabilities. Despite these merits, concerns persist regarding the security of
LoRa networks, especially in situations where device identification and
authentication are imperative to secure the reliable access to the LoRa
networks. This paper explores a deep learning (DL) approach to tackle these
concerns, focusing on two critical tasks, namely (i) identifying LoRa devices
and (ii) classifying them to legitimate and rogue devices. Deep neural networks
(DNNs), encompassing both convolutional and feedforward neural networks, are
trained for these tasks using actual LoRa signal data. In this setting, the
adversaries may spoof rogue LoRa signals through the kernel density estimation
(KDE) method based on legitimate device signals that are received by the
adversaries. Two cases are considered, (i) training two separate classifiers,
one for each of the two tasks, and (ii) training a multi-task classifier for
both tasks. The vulnerabilities of the resulting DNNs to manipulations in input
samples are studied in form of untargeted and targeted adversarial attacks
using the Fast Gradient Sign Method (FGSM). Individual and common perturbations
are considered against single-task and multi-task classifiers for the LoRa
signal analysis. To provide resilience against such attacks, a defense approach
is presented by increasing the robustness of classifiers with adversarial
training. Results quantify how vulnerable LoRa signal classification tasks are
to adversarial attacks and emphasize the need to fortify IoT applications
against these subtle yet effective threats.


http://arxiv.org/abs/2412.20768
Sample Correlation for Fingerprinting Deep Face Recognition. (98%)
Jiyang Guan; Jian Liang; Yanbo Wang; Ran He
  Face recognition has witnessed remarkable advancements in recent years,
thanks to the development of deep learning techniques.However, an off-the-shelf
face recognition model as a commercial service could be stolen by model
stealing attacks, posing great threats to the rights of the model owner.Model
fingerprinting, as a model stealing detection method, aims to verify whether a
suspect model is stolen from the victim model, gaining more and more attention
nowadays.Previous methods always utilize transferable adversarial examples as
the model fingerprint, but this method is known to be sensitive to adversarial
defense and transfer learning techniques.To address this issue, we consider the
pairwise relationship between samples instead and propose a novel yet simple
model stealing detection method based on SAmple Correlation (SAC).Specifically,
we present SAC-JC that selects JPEG compressed samples as model inputs and
calculates the correlation matrix among their model outputs.Extensive results
validate that SAC successfully defends against various model stealing attacks
in deep face recognition, encompassing face verification and face emotion
recognition, exhibiting the highest performance in terms of AUC, p-value and F1
score.Furthermore, we extend our evaluation of SAC-JC to object recognition
datasets including Tiny-ImageNet and CIFAR10, which also demonstrates the
superior performance of SAC-JC to previous methods.The code will be available
at \url{https://github.com/guanjiyang/SAC_JC}.


http://arxiv.org/abs/2412.20807
Two Heads Are Better Than One: Averaging along Fine-Tuning to Improve Targeted Transferability. (93%)
Hui Zeng; Sanshuai Cui; Biwei Chen; Anjie Peng
  With much longer optimization time than that of untargeted attacks
notwithstanding, the transferability of targeted attacks is still far from
satisfactory. Recent studies reveal that fine-tuning an existing adversarial
example (AE) in feature space can efficiently boost its targeted
transferability. However, existing fine-tuning schemes only utilize the
endpoint and ignore the valuable information in the fine-tuning trajectory.
Noting that the vanilla fine-tuning trajectory tends to oscillate around the
periphery of a flat region of the loss surface, we propose averaging over the
fine-tuning trajectory to pull the crafted AE towards a more centered region.
We compare the proposed method with existing fine-tuning schemes by integrating
them with state-of-the-art targeted attacks in various attacking scenarios.
Experimental results uphold the superiority of the proposed method in boosting
targeted transferability. The code is available at github.com/zengh5/Avg_FT.


http://arxiv.org/abs/2412.20756
Unsupervised dense retrieval with conterfactual contrastive learning. (54%)
Haitian Chen; Qingyao Ai; Xiao Wang; Yiqun Liu; Fen Lin; Qin Liu
  Efficiently retrieving a concise set of candidates from a large document
corpus remains a pivotal challenge in Information Retrieval (IR). Neural
retrieval models, particularly dense retrieval models built with transformers
and pretrained language models, have been popular due to their superior
performance. However, criticisms have also been raised on their lack of
explainability and vulnerability to adversarial attacks. In response to these
challenges, we propose to improve the robustness of dense retrieval models by
enhancing their sensitivity of fine-graned relevance signals. A model achieving
sensitivity in this context should exhibit high variances when documents' key
passages determining their relevance to queries have been modified, while
maintaining low variances for other changes in irrelevant passages. This
sensitivity allows a dense retrieval model to produce robust results with
respect to attacks that try to promote documents without actually increasing
their relevance. It also makes it possible to analyze which part of a document
is actually relevant to a query, and thus improve the explainability of the
retrieval model. Motivated by causality and counterfactual analysis, we propose
a series of counterfactual regularization methods based on game theory and
unsupervised learning with counterfactual passages. Experiments show that, our
method can extract key passages without reliance on the passage-level relevance
annotations. Moreover, the regularized dense retrieval models exhibit
heightened robustness against adversarial attacks, surpassing the
state-of-the-art anti-attack methods.


http://arxiv.org/abs/2412.20953
GASLITEing the Retrieval: Exploring Vulnerabilities in Dense Embedding-based Search. (50%)
Matan Ben-Tov; Mahmood Sharif
  Dense embedding-based text retrieval$\unicode{x2013}$retrieval of relevant
passages from corpora via deep learning encodings$\unicode{x2013}$has emerged
as a powerful method attaining state-of-the-art search results and popularizing
the use of Retrieval Augmented Generation (RAG). Still, like other search
methods, embedding-based retrieval may be susceptible to search-engine
optimization (SEO) attacks, where adversaries promote malicious content by
introducing adversarial passages to corpora. To faithfully assess and gain
insights into the susceptibility of such systems to SEO, this work proposes the
GASLITE attack, a mathematically principled gradient-based search method for
generating adversarial passages without relying on the corpus content or
modifying the model. Notably, GASLITE's passages (1) carry adversary-chosen
information while (2) achieving high retrieval ranking for a selected query
distribution when inserted to corpora. We use GASLITE to extensively evaluate
retrievers' robustness, testing nine advanced models under varied threat
models, while focusing on realistic adversaries targeting queries on a specific
concept (e.g., a public figure). We found GASLITE consistently outperformed
baselines by $\geq$140% success rate, in all settings. Particularly,
adversaries using GASLITE require minimal effort to manipulate search
results$\unicode{x2013}$by injecting a negligible amount of adversarial
passages ($\leq$0.0001% of the corpus), they could make them visible in the
top-10 results for 61-100% of unseen concept-specific queries against most
evaluated models. Inspecting variance in retrievers' robustness, we identify
key factors that may contribute to models' susceptibility to SEO, including
specific properties in the embedding space's geometry.


http://arxiv.org/abs/2412.21016
Automated Robustness Testing for LLM-based NLP Software. (2%)
Mingxuan Xiao; Yan Xiao; Shunhui Ji; Hanbo Cai; Lei Xue; Pengcheng Zhang
  Benefiting from the advancements in LLMs, NLP software has undergone rapid
development. Such software is widely employed in various safety-critical tasks,
such as financial sentiment analysis, toxic content moderation, and log
generation. To our knowledge, there are no known automated robustness testing
methods specifically designed for LLM-based NLP software. Given the complexity
of LLMs and the unpredictability of real-world inputs (including prompts and
examples), it is essential to examine the robustness of overall inputs to
ensure the safety of such software.
  To this end, this paper introduces the first AutOmated Robustness Testing
frAmework, AORTA, which reconceptualizes the testing process into a
combinatorial optimization problem. Existing testing methods designed for
DNN-based software can be applied to LLM-based software by AORTA, but their
effectiveness is limited. To address this, we propose a novel testing method
for LLM-based software within AORTA called Adaptive Beam Search. ABS is
tailored for the expansive feature space of LLMs and improves testing
effectiveness through an adaptive beam width and the capability for
backtracking.
  We successfully embed 18 test methods in the designed framework AORTA and
compared the test validity of ABS with three datasets and five threat models.
ABS facilitates a more comprehensive and accurate robustness assessment before
software deployment, with an average test success rate of 86.138%. Compared to
the currently best-performing baseline PWWS, ABS significantly reduces the
computational overhead by up to 3441.895 seconds per successful test case and
decreases the number of queries by 218.762 times on average. Furthermore, test
cases generated by ABS exhibit greater naturalness and transferability.


http://arxiv.org/abs/2412.20804
DELA: A Novel Approach for Detecting Errors Induced by Large Atomic Condition Numbers. (1%)
Youshuai Tan; Zhanwei Zhang; Jinfu Chen; Zishuo Ding; Jifeng Xuan; Weiyi Shang
  Numerical programs form the foundation of modern science and engineering,
providing essential solutions to complex mathematical problems. Therefore,
errors in numerical results would lead to harmful consequences, especially in
safety-critical applications. Since only a few inputs may lead to substantial
errors for numerical programs, it is essential to determine whether a given
input could result in a significant error. Existing researchers tend to use the
results of high-precision programs to assess whether there is a substantial
error, which introduces three main challenges: difficulty of implementation,
existence of potential faults in the detection of numerical errors, and long
execution time.
  To address these limitations, we propose a novel approach named DELA. Our
approach is based on the observation that most numerical errors stem from large
condition numbers in atomic operations (such as subtraction), which then
propagate and accumulate. DELA injects small perturbations into the results of
individual atomic operations within the program and compares the outcomes of
the original program with the perturbed version to detect errors. We evaluate
DELA with datasets from ATOMU and HSED, as well as data from a complex linear
system-solving program. Experimental results demonstrate that we can detect all
the significant errors that were reported by prior research. DELA shows strong
alignment with high-precision programs of ATOMU and HSED, with average Pearson
and Spearman correlations of 0.86 and 0.61. Additionally, DELA effectively
detects significant errors in complex programs, achieving correlation scores of
0.9763 and 0.8993. More importantly, in experiments with ATOMU and HSED, DELA's
perturbed programs run within only 0.13% of the time needed by high-precision
versions; while for the linear system-solving programs, DELA is 73.46 times
faster than the high-precision programs.


http://arxiv.org/abs/2412.20392
Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuning. (75%)
Zhifang Zhang; Shuo He; Bingquan Shen; Lei Feng
  Multimodal contrastive learning models (e.g., CLIP) can learn high-quality
representations from large-scale image-text datasets, yet they exhibit
significant vulnerabilities to backdoor attacks, raising serious safety
concerns. In this paper, we disclose that CLIP's vulnerabilities primarily stem
from its excessive encoding of class-irrelevant features, which can compromise
the model's visual feature resistivity to input perturbations, making it more
susceptible to capturing the trigger patterns inserted by backdoor attacks.
Inspired by this finding, we propose Repulsive Visual Prompt Tuning (RVPT), a
novel defense approach that employs specially designed deep visual prompt
tuning and feature-repelling loss to eliminate excessive class-irrelevant
features while simultaneously optimizing cross-entropy loss to maintain clean
accuracy. Unlike existing multimodal backdoor defense methods that typically
require the availability of poisoned data or involve fine-tuning the entire
model, RVPT leverages few-shot downstream clean samples and only tunes a small
number of parameters. Empirical results demonstrate that RVPT tunes only 0.27\%
of the parameters relative to CLIP, yet it significantly outperforms
state-of-the-art baselines, reducing the attack success rate from 67.53\% to
2.76\% against SoTA attacks and effectively generalizing its defensive
capabilities across multiple datasets.


http://arxiv.org/abs/2501.00066
On Adversarial Robustness of Language Models in Transfer Learning. (10%)
Bohdan Turbal; Anastasiia Mazur; Jiaxu Zhao; Mykola Pechenizkiy
  We investigate the adversarial robustness of LLMs in transfer learning
scenarios. Through comprehensive experiments on multiple datasets (MBIB Hate
Speech, MBIB Political Bias, MBIB Gender Bias) and various model architectures
(BERT, RoBERTa, GPT-2, Gemma, Phi), we reveal that transfer learning, while
improving standard performance metrics, often leads to increased vulnerability
to adversarial attacks. Our findings demonstrate that larger models exhibit
greater resilience to this phenomenon, suggesting a complex interplay between
model size, architecture, and adaptation methods. Our work highlights the
crucial need for considering adversarial robustness in transfer learning
scenarios and provides insights into maintaining model security without
compromising performance. These findings have significant implications for the
development and deployment of LLMs in real-world applications where both
performance and robustness are paramount.


http://arxiv.org/abs/2412.20670
Prototypical Distillation and Debiased Tuning for Black-box Unsupervised Domain Adaptation. (10%)
Jian Liang; Lijun Sheng; Hongmin Liu; Ran He
  Unsupervised domain adaptation aims to transfer knowledge from a related,
label-rich source domain to an unlabeled target domain, thereby circumventing
the high costs associated with manual annotation. Recently, there has been
growing interest in source-free domain adaptation, a paradigm in which only a
pre-trained model, rather than the labeled source data, is provided to the
target domain. Given the potential risk of source data leakage via model
inversion attacks, this paper introduces a novel setting called black-box
domain adaptation, where the source model is accessible only through an API
that provides the predicted label along with the corresponding confidence value
for each query. We develop a two-step framework named $\textbf{Pro}$totypical
$\textbf{D}$istillation and $\textbf{D}$ebiased tun$\textbf{ing}$
($\textbf{ProDDing}$). In the first step, ProDDing leverages both the raw
predictions from the source model and prototypes derived from the target domain
as teachers to distill a customized target model. In the second step, ProDDing
keeps fine-tuning the distilled model by penalizing logits that are biased
toward certain classes. Empirical results across multiple benchmarks
demonstrate that ProDDing outperforms existing black-box domain adaptation
methods. Moreover, in the case of hard-label black-box domain adaptation, where
only predicted labels are available, ProDDing achieves significant improvements
over these methods. Code will be available at
\url{https://github.com/tim-learn/ProDDing/}.


http://arxiv.org/abs/2412.20529
Attacks on the neural network and defense methods. (2%)
A. Korenev; G. Belokrylov; B. Lodonova; A. Novokhrestov
  This article will discuss the use of attacks on a neural network trained on
audio data, as well as possible methods of protection against these attacks.
FGSM, PGD and CW attacks, as well as data poisoning, will be considered. Within
the framework of protection, Art-IBM and advertorch libraries will be
considered. The obtained accuracy metrics within the framework of attack
applications are presented


http://arxiv.org/abs/2412.20476
Cut the Deadwood Out: Post-Training Model Purification with Selective Module Substitution. (2%)
Yao Tong; Weijun Li; Xuanli He; Haolan Zhan; Qiongkai Xu
  The success of DNNs often depends on training with large-scale datasets, but
building such datasets is both expensive and challenging. Consequently, public
datasets from open-source platforms like HuggingFace have become popular,
posing significant risks of data poisoning attacks. Existing backdoor defenses
in NLP primarily focus on identifying and removing poisoned samples; however,
purifying a backdoored model with these sample-cleaning approaches typically
requires expensive retraining. Therefore, we propose Greedy Module Substitution
(GMS), which identifies and substitutes ''deadwood'' modules (i.e., components
critical to backdoor pathways) in a backdoored model to purify it. Our method
relaxes the common dependency of prior model purification methods on clean
datasets or clean auxiliary models. When applied to RoBERTa-large under
backdoor attacks, GMS demonstrates strong effectiveness across various
settings, particularly against widely recognized challenging attacks like LWS,
achieving a post-purification attack success rate (ASR) of 9.7% on SST-2
compared to 58.8% for the best baseline approach.


http://arxiv.org/abs/2412.20025
A Robust Adversarial Ensemble with Causal (Feature Interaction) Interpretations for Image Classification. (99%)
Chunheng Zhao; Pierluigi Pisu; Gurcan Comert; Negash Begashaw; Varghese Vaidyan; Nina Christine Hubig
  Deep learning-based discriminative classifiers, despite their remarkable
success, remain vulnerable to adversarial examples that can mislead model
predictions. While adversarial training can enhance robustness, it fails to
address the intrinsic vulnerability stemming from the opaque nature of these
black-box models. We present a deep ensemble model that combines discriminative
features with generative models to achieve both high accuracy and adversarial
robustness. Our approach integrates a bottom-level pre-trained discriminative
network for feature extraction with a top-level generative classification
network that models adversarial input distributions through a deep latent
variable model. Using variational Bayes, our model achieves superior robustness
against white-box adversarial attacks without adversarial training. Extensive
experiments on CIFAR-10 and CIFAR-100 demonstrate our model's superior
adversarial robustness. Through evaluations using counterfactual metrics and
feature interaction-based metrics, we establish correlations between model
interpretability and adversarial robustness. Additionally, preliminary results
on Tiny-ImageNet validate our approach's scalability to more complex datasets,
offering a practical solution for developing robust image classification
models.


http://arxiv.org/abs/2412.20087
On the Validity of Traditional Vulnerability Scoring Systems for Adversarial Attacks against LLMs. (13%)
Atmane Ayoub Mansour Bahar; Ahmad Samer Wazan
  This research investigates the effectiveness of established vulnerability
metrics, such as the Common Vulnerability Scoring System (CVSS), in evaluating
attacks against Large Language Models (LLMs), with a focus on Adversarial
Attacks (AAs). The study explores the influence of both general and specific
metric factors in determining vulnerability scores, providing new perspectives
on potential enhancements to these metrics.
  This study adopts a quantitative approach, calculating and comparing the
coefficient of variation of vulnerability scores across 56 adversarial attacks
on LLMs. The attacks, sourced from various research papers, and obtained
through online databases, were evaluated using multiple vulnerability metrics.
Scores were determined by averaging the values assessed by three distinct LLMs.
The results indicate that existing scoring-systems yield vulnerability scores
with minimal variation across different attacks, suggesting that many of the
metric factors are inadequate for assessing adversarial attacks on LLMs. This
is particularly true for context-specific factors or those with predefined
value sets, such as those in CVSS. These findings support the hypothesis that
current vulnerability metrics, especially those with rigid values, are limited
in evaluating AAs on LLMs, highlighting the need for the development of more
flexible, generalized metrics tailored to such attacks.
  This research offers a fresh analysis of the effectiveness and applicability
of established vulnerability metrics, particularly in the context of
Adversarial Attacks on Large Language Models, both of which have gained
significant attention in recent years. Through extensive testing and
calculations, the study underscores the limitations of these metrics and opens
up new avenues for improving and refining vulnerability assessment frameworks
specifically tailored for LLMs.


http://arxiv.org/abs/2412.20086
MAFT: Efficient Model-Agnostic Fairness Testing for Deep Neural Networks via Zero-Order Gradient Search. (9%)
Zhaohui Wang; Min Zhang; Jingran Yang; Bojie Shao; Min Zhang
  Deep neural networks (DNNs) have shown powerful performance in various
applications and are increasingly being used in decision-making systems.
However, concerns about fairness in DNNs always persist. Some efficient
white-box fairness testing methods about individual fairness have been
proposed. Nevertheless, the development of black-box methods has stagnated, and
the performance of existing methods is far behind that of white-box methods. In
this paper, we propose a novel black-box individual fairness testing method
called Model-Agnostic Fairness Testing (MAFT). By leveraging MAFT,
practitioners can effectively identify and address discrimination in DL models,
regardless of the specific algorithm or architecture employed. Our approach
adopts lightweight procedures such as gradient estimation and attribute
perturbation rather than non-trivial procedures like symbol execution,
rendering it significantly more scalable and applicable than existing methods.
We demonstrate that MAFT achieves the same effectiveness as state-of-the-art
white-box methods whilst improving the applicability to large-scale networks.
Compared to existing black-box approaches, our approach demonstrates
distinguished performance in discovering fairness violations w.r.t
effectiveness (approximately 14.69 times) and efficiency (approximately 32.58
times).


http://arxiv.org/abs/2501.00055
LLM-Virus: Evolutionary Jailbreak Attack on Large Language Models. (2%)
Miao Yu; Junfeng Fang; Yingjie Zhou; Xing Fan; Kun Wang; Shirui Pan; Qingsong Wen
  While safety-aligned large language models (LLMs) are increasingly used as
the cornerstone for powerful systems such as multi-agent frameworks to solve
complex real-world problems, they still suffer from potential adversarial
queries, such as jailbreak attacks, which attempt to induce harmful content.
Researching attack methods allows us to better understand the limitations of
LLM and make trade-offs between helpfulness and safety. However, existing
jailbreak attacks are primarily based on opaque optimization techniques (e.g.
token-level gradient descent) and heuristic search methods like LLM refinement,
which fall short in terms of transparency, transferability, and computational
cost. In light of these limitations, we draw inspiration from the evolution and
infection processes of biological viruses and propose LLM-Virus, a jailbreak
attack method based on evolutionary algorithm, termed evolutionary jailbreak.
LLM-Virus treats jailbreak attacks as both an evolutionary and transfer
learning problem, utilizing LLMs as heuristic evolutionary operators to ensure
high attack efficiency, transferability, and low time cost. Our experimental
results on multiple safety benchmarks show that LLM-Virus achieves competitive
or even superior performance compared to existing attack methods.


http://arxiv.org/abs/2412.20154
Defending Against Network Attacks for Secure AI Agent Migration in Vehicular Metaverses. (1%)
Xinru Wen; Jinbo Wen; Ming Xiao; Jiawen Kang; Tao Zhang; Xiaohuan Li; Chuanxi Chen; Dusit Niyato
  Vehicular metaverses, blending traditional vehicular networks with metaverse
technology, are expected to revolutionize fields such as autonomous driving. As
virtual intelligent assistants in vehicular metaverses, Artificial Intelligence
(AI) agents powered by large language models can create immersive 3D virtual
spaces for passengers to enjoy on-broad vehicular applications and services. To
provide users with seamless and engaging virtual interactions, resource-limited
vehicles offload AI agents to RoadSide Units (RSUs) with adequate communication
and computational capabilities. Due to the mobility of vehicles and the limited
coverage of RSUs, AI agents need to migrate from one RSU to another RSU.
However, potential network attacks pose significant challenges to ensuring
reliable and efficient AI agent migration. In this paper, we first explore
specific network attacks including traffic-based attacks (i.e., DDoS attacks)
and infrastructure-based attacks (i.e., malicious RSU attacks). Then, we model
the AI agent migration process as a Partially Observable Markov Decision
Process (POMDP) and apply multi-agent proximal policy optimization algorithms
to mitigate DDoS attacks. In addition, we propose a trust assessment mechanism
to counter malicious RSU attacks. Numerical results validate that the proposed
solutions effectively defend against these network attacks and reduce the total
latency of AI agent migration by approximately 43.3%.


http://arxiv.org/abs/2412.19947
Standard-Deviation-Inspired Regularization for Improving Adversarial Robustness. (99%)
Olukorede Fakorede; Modeste Atsague; Jin Tian
  Adversarial Training (AT) has been demonstrated to improve the robustness of
deep neural networks (DNNs) against adversarial attacks. AT is a min-max
optimization procedure where in adversarial examples are generated to train a
more robust DNN. The inner maximization step of AT increases the losses of
inputs with respect to their actual classes. The outer minimization involves
minimizing the losses on the adversarial examples obtained from the inner
maximization. This work proposes a standard-deviation-inspired (SDI)
regularization term to improve adversarial robustness and generalization. We
argue that the inner maximization in AT is similar to minimizing a modified
standard deviation of the model's output probabilities. Moreover, we suggest
that maximizing this modified standard deviation can complement the outer
minimization of the AT framework. To support our argument, we experimentally
show that the SDI measure can be used to craft adversarial examples.
Additionally, we demonstrate that combining the SDI regularization term with
existing AT variants enhances the robustness of DNNs against stronger attacks,
such as CW and Auto-attack, and improves generalization.


http://arxiv.org/abs/2412.19747
Enhancing Adversarial Robustness of Deep Neural Networks Through Supervised Contrastive Learning. (96%)
Longwei Wang; Navid Nayyem; Abdullah Rakin
  Adversarial attacks exploit the vulnerabilities of convolutional neural
networks by introducing imperceptible perturbations that lead to
misclassifications, exposing weaknesses in feature representations and decision
boundaries. This paper presents a novel framework combining supervised
contrastive learning and margin-based contrastive loss to enhance adversarial
robustness. Supervised contrastive learning improves the structure of the
feature space by clustering embeddings of samples within the same class and
separating those from different classes. Margin-based contrastive loss,
inspired by support vector machines, enforces explicit constraints to create
robust decision boundaries with well-defined margins. Experiments on the
CIFAR-100 dataset with a ResNet-18 backbone demonstrate robustness performance
improvements in adversarial accuracy under Fast Gradient Sign Method attacks.


http://arxiv.org/abs/2412.19523
Attribution for Enhanced Explanation with Transferable Adversarial eXploration. (92%)
Zhiyu Zhu; Jiayu Zhang; Zhibo Jin; Huaming Chen; Jianlong Zhou; Fang Chen
  The interpretability of deep neural networks is crucial for understanding
model decisions in various applications, including computer vision.
AttEXplore++, an advanced framework built upon AttEXplore, enhances attribution
by incorporating transferable adversarial attack methods such as MIG and GRA,
significantly improving the accuracy and robustness of model explanations. We
conduct extensive experiments on five models, including CNNs (Inception-v3,
ResNet-50, VGG16) and vision transformers (MaxViT-T, ViT-B/16), using the
ImageNet dataset. Our method achieves an average performance improvement of
7.57\% over AttEXplore and 32.62\% compared to other state-of-the-art
interpretability algorithms. Using insertion and deletion scores as evaluation
metrics, we show that adversarial transferability plays a vital role in
enhancing attribution results. Furthermore, we explore the impact of
randomness, perturbation rate, noise amplitude, and diversity probability on
attribution performance, demonstrating that AttEXplore++ provides more stable
and reliable explanations across various models. We release our code at:
https://anonymous.4open.science/r/ATTEXPLOREP-8435/


http://arxiv.org/abs/2412.20006
Adversarial Robustness for Deep Learning-based Wildfire Detection Models. (92%)
Ryo Ide; Lei Yang
  Smoke detection using Deep Neural Networks (DNNs) is an effective approach
for early wildfire detection. However, because smoke is temporally and
spatially anomalous, there are limitations in collecting sufficient training
data. This raises overfitting and bias concerns in existing DNN-based wildfire
detection models. Thus, we introduce WARP (Wildfire Adversarial Robustness
Procedure), the first model-agnostic framework for evaluating the adversarial
robustness of DNN-based wildfire detection models. WARP addresses limitations
in smoke image diversity using global and local adversarial attack methods. The
global attack method uses image-contextualized Gaussian noise, while the local
attack method uses patch noise injection, tailored to address critical aspects
of wildfire detection. Leveraging WARP's model-agnostic capabilities, we assess
the adversarial robustness of real-time Convolutional Neural Networks (CNNs)
and Transformers. The analysis revealed valuable insights into the models'
limitations. Specifically, the global attack method demonstrates that the
Transformer model has more than 70\% precision degradation than the CNN against
global noise. In contrast, the local attack method shows that both models are
susceptible to cloud image injections when detecting smoke-positive instances,
suggesting a need for model improvements through data augmentation. WARP's
comprehensive robustness analysis contributed to the development of
wildfire-specific data augmentation strategies, marking a step toward
practicality.


http://arxiv.org/abs/2412.19354
Federated Hybrid Training and Self-Adversarial Distillation: Towards Robust Edge Networks. (45%)
Yu Qiao; Apurba Adhikary; Kitae Kim; Eui-Nam Huh; Zhu Han; Choong Seon Hong
  Federated learning (FL) is a distributed training technology that enhances
data privacy in mobile edge networks by allowing data owners to collaborate
without transmitting raw data to the edge server. However, data heterogeneity
and adversarial attacks pose challenges to develop an unbiased and robust
global model for edge deployment. To address this, we propose Federated hyBrid
Adversarial training and self-adversarial disTillation (FedBAT), a new
framework designed to improve both robustness and generalization of the global
model. FedBAT seamlessly integrates hybrid adversarial training and
self-adversarial distillation into the conventional FL framework from data
augmentation and feature distillation perspectives. From a data augmentation
perspective, we propose hybrid adversarial training to defend against
adversarial attacks by balancing accuracy and robustness through a weighted
combination of standard and adversarial training. From a feature distillation
perspective, we introduce a novel augmentation-invariant adversarial
distillation method that aligns local adversarial features of augmented images
with their corresponding unbiased global clean features. This alignment can
effectively mitigate bias from data heterogeneity while enhancing both the
robustness and generalization of the global model. Extensive experimental
results across multiple datasets demonstrate that FedBAT yields comparable or
superior performance gains in improving robustness while maintaining accuracy
compared to several baselines.


http://arxiv.org/abs/2412.19394
An Engorgio Prompt Makes Large Language Model Babble on. (15%)
Jianshuo Dong; Ziyuan Zhang; Qingjie Zhang; Han Qiu; Tianwei Zhang; Hao Wang; Hewu Li; Qi Li; Chao Zhang; Ke Xu
  Auto-regressive large language models (LLMs) have yielded impressive
performance in many real-world tasks. However, the new paradigm of these LLMs
also exposes novel threats. In this paper, we explore their vulnerability to
inference cost attacks, where a malicious user crafts Engorgio prompts to
intentionally increase the computation cost and latency of the inference
process. We design Engorgio, a novel methodology, to efficiently generate
adversarial Engorgio prompts to affect the target LLM's service availability.
Engorgio has the following two technical contributions. (1) We employ a
parameterized distribution to track LLMs' prediction trajectory. (2) Targeting
the auto-regressive nature of LLMs' inference process, we propose novel loss
functions to stably suppress the appearance of the <EOS> token, whose
occurrence will interrupt the LLM's generation process. We conduct extensive
experiments on 13 open-sourced LLMs with parameters ranging from 125M to 30B.
The results show that Engorgio prompts can successfully induce LLMs to generate
abnormally long outputs (i.e., roughly 2-13$\times$ longer to reach 90%+ of the
output length limit) in a white-box scenario and our real-world experiment
demonstrates Engergio's threat to LLM service with limited computing resources.
The code is accessible at https://github.com/jianshuod/Engorgio-prompt.


http://arxiv.org/abs/2412.19311
xSRL: Safety-Aware Explainable Reinforcement Learning -- Safety as a Product of Explainability. (2%)
Risal Shahriar Shefin; Md Asifur Rahman; Thai Le; Sarra Alqahtani
  Reinforcement learning (RL) has shown great promise in simulated
environments, such as games, where failures have minimal consequences. However,
the deployment of RL agents in real-world systems such as autonomous vehicles,
robotics, UAVs, and medical devices demands a higher level of safety and
transparency, particularly when facing adversarial threats. Safe RL algorithms
have been developed to address these concerns by optimizing both task
performance and safety constraints. However, errors are inevitable, and when
they occur, it is essential that the RL agents can also explain their actions
to human operators. This makes trust in the safety mechanisms of RL systems
crucial for effective deployment. Explainability plays a key role in building
this trust by providing clear, actionable insights into the agent's
decision-making process, ensuring that safety-critical decisions are well
understood. While machine learning (ML) has seen significant advances in
interpretability and visualization, explainability methods for RL remain
limited. Current tools fail to address the dynamic, sequential nature of RL and
its needs to balance task performance with safety constraints over time. The
re-purposing of traditional ML methods, such as saliency maps, is inadequate
for safety-critical RL applications where mistakes can result in severe
consequences. To bridge this gap, we propose xSRL, a framework that integrates
both local and global explanations to provide a comprehensive understanding of
RL agents' behavior. xSRL also enables developers to identify policy
vulnerabilities through adversarial attacks, offering tools to debug and patch
agents without retraining. Our experiments and user studies demonstrate xSRL's
effectiveness in increasing safety in RL systems, making them more reliable and
trustworthy for real-world deployment. Code is available at
https://github.com/risal-shefin/xSRL.


http://arxiv.org/abs/2412.18815
Distortion-Aware Adversarial Attacks on Bounding Boxes of Object Detectors. (99%)
Pham Phuc; Son Vuong; Khang Nguyen; Tuan Dang
  Deep learning-based object detection has become ubiquitous in the last decade
due to its high accuracy in many real-world applications. With this growing
trend, these models are interested in being attacked by adversaries, with most
of the results being on classifiers, which do not match the context of
practical object detection. In this work, we propose a novel method to fool
object detectors, expose the vulnerability of state-of-the-art detectors, and
promote later works to build more robust detectors to adversarial examples. Our
method aims to generate adversarial images by perturbing object confidence
scores during training, which is crucial in predicting confidence for each
class in the testing phase. Herein, we provide a more intuitive technique to
embed additive noises based on detected objects' masks and the training loss
with distortion control over the original image by leveraging the gradient of
iterative images. To verify the proposed method, we perform adversarial attacks
against different object detectors, including the most recent state-of-the-art
models like YOLOv8, Faster R-CNN, RetinaNet, and Swin Transformer. We also
evaluate our technique on MS COCO 2017 and PASCAL VOC 2012 datasets and analyze
the trade-off between success attack rate and image distortion. Our experiments
show that the achievable success attack rate is up to $100$\% and up to $98$\%
when performing white-box and black-box attacks, respectively. The source code
and relevant documentation for this work are available at the following link:
https://github.com/anonymous20210106/attack_detector


http://arxiv.org/abs/2412.18844
Improving Integrated Gradient-based Transferable Adversarial Examples by Refining the Integration Path. (98%)
Yuchen Ren; Zhengyu Zhao; Chenhao Lin; Bo Yang; Lu Zhou; Zhe Liu; Chao Shen
  Transferable adversarial examples are known to cause threats in practical,
black-box attack scenarios. A notable approach to improving transferability is
using integrated gradients (IG), originally developed for model
interpretability. In this paper, we find that existing IG-based attacks have
limited transferability due to their naive adoption of IG in model
interpretability. To address this limitation, we focus on the IG integration
path and refine it in three aspects: multiplicity, monotonicity, and diversity,
supported by theoretical analyses. We propose the Multiple Monotonic
Diversified Integrated Gradients (MuMoDIG) attack, which can generate highly
transferable adversarial examples on different CNN and ViT models and defenses.
Experiments validate that MuMoDIG outperforms the latest IG-based attack by up
to 37.3\% and other state-of-the-art attacks by 8.4\%. In general, our study
reveals that migrating established techniques to improve transferability may
require non-trivial efforts. Code is available at
\url{https://github.com/RYC-98/MuMoDIG}.


http://arxiv.org/abs/2412.19015
Imperceptible Adversarial Attacks on Point Clouds Guided by Point-to-Surface Field. (83%)
Keke Tang; Weiyao Ke; Weilong Peng; Xiaofei Wang; Ziyong Du; Zhize Wu; Peican Zhu; Zhihong Tian
  Adversarial attacks on point clouds are crucial for assessing and improving
the adversarial robustness of 3D deep learning models. Traditional solutions
strictly limit point displacement during attacks, making it challenging to
balance imperceptibility with adversarial effectiveness. In this paper, we
attribute the inadequate imperceptibility of adversarial attacks on point
clouds to deviations from the underlying surface. To address this, we introduce
a novel point-to-surface (P2S) field that adjusts adversarial perturbation
directions by dragging points back to their original underlying surface.
Specifically, we use a denoising network to learn the gradient field of the
logarithmic density function encoding the shape's surface, and apply a
distance-aware adjustment to perturbation directions during attacks, thereby
enhancing imperceptibility. Extensive experiments show that adversarial attacks
guided by our P2S field are more imperceptible, outperforming state-of-the-art
methods.


http://arxiv.org/abs/2412.18886
Adversarial Training for Graph Neural Networks via Graph Subspace Energy Optimization. (75%)
Ganlin Liu; Ziling Liang; Xiaowei Huang; Xinping Yi; Shi Jin
  Despite impressive capability in learning over graph-structured data, graph
neural networks (GNN) suffer from adversarial topology perturbation in both
training and inference phases. While adversarial training has demonstrated
remarkable effectiveness in image classification tasks, its suitability for GNN
models has been doubted until a recent advance that shifts the focus from
transductive to inductive learning. Still, GNN robustness in the inductive
setting is under-explored, and it calls for deeper understanding of GNN
adversarial training. To this end, we propose a new concept of graph subspace
energy (GSE) -- a generalization of graph energy that measures graph stability
-- of the adjacency matrix, as an indicator of GNN robustness against topology
perturbations. To further demonstrate the effectiveness of such concept, we
propose an adversarial training method with the perturbed graphs generated by
maximizing the GSE regularization term, referred to as AT-GSE. To deal with the
local and global topology perturbations raised respectively by LRBCD and PRBCD,
we employ randomized SVD (RndSVD) and Nystrom low-rank approximation to favor
the different aspects of the GSE terms. An extensive set of experiments shows
that AT-GSE outperforms consistently the state-of-the-art GNN adversarial
training methods over different homophily and heterophily datasets in terms of
adversarial accuracy, whilst more surprisingly achieving a superior clean
accuracy on non-perturbed graphs.


http://arxiv.org/abs/2412.18952
Bridging Interpretability and Robustness Using LIME-Guided Model Refinement. (69%)
Navid Nayyem; Abdullah Rakin; Longwei Wang
  This paper explores the intricate relationship between interpretability and
robustness in deep learning models. Despite their remarkable performance across
various tasks, deep learning models often exhibit critical vulnerabilities,
including susceptibility to adversarial attacks, over-reliance on spurious
correlations, and a lack of transparency in their decision-making processes. To
address these limitations, we propose a novel framework that leverages Local
Interpretable Model-Agnostic Explanations (LIME) to systematically enhance
model robustness. By identifying and mitigating the influence of irrelevant or
misleading features, our approach iteratively refines the model, penalizing
reliance on these features during training. Empirical evaluations on multiple
benchmark datasets demonstrate that LIME-guided refinement not only improves
interpretability but also significantly enhances resistance to adversarial
perturbations and generalization to out-of-distribution data.


http://arxiv.org/abs/2412.18975
Injecting Bias into Text Classification Models using Backdoor Attacks. (15%)
A. Dilara Yavuz; M. Emre Gursoy
  The rapid growth of natural language processing (NLP) and pre-trained
language models have enabled accurate text classification in a variety of
settings. However, text classification models are susceptible to backdoor
attacks, where an attacker embeds a trigger into the victim model to make the
model predict attacker-desired labels in targeted scenarios. In this paper, we
propose to utilize backdoor attacks for a new purpose: bias injection. We
develop a backdoor attack in which a subset of the training dataset is poisoned
to associate strong male actors with negative sentiment. We execute our attack
on two popular text classification datasets (IMDb and SST) and seven different
models ranging from traditional Doc2Vec-based models to LSTM networks and
modern transformer-based BERT and RoBERTa models. Our results show that the
reduction in backdoored models' benign classification accuracy is limited,
implying that our attacks remain stealthy, whereas the models successfully
learn to associate strong male actors with negative sentiment (100% attack
success rate with >= 3% poison rate). Attacks on BERT and RoBERTa are
particularly more stealthy and effective, demonstrating an increased risk of
using modern and larger models. We also measure the generalizability of our
bias injection by proposing two metrics: (i) U-BBSR which uses previously
unseen words when measuring attack success, and (ii) P-BBSR which measures
attack success using paraphrased test samples. U-BBSR and P-BBSR results show
that the bias injected by our attack can go beyond memorizing a trigger phrase.


http://arxiv.org/abs/2412.18791
Protective Perturbations against Unauthorized Data Usage in Diffusion-based Image Generation. (10%)
Sen Peng; Jijia Yang; Mingyue Wang; Jianfei He; Xiaohua Jia
  Diffusion-based text-to-image models have shown immense potential for various
image-related tasks. However, despite their prominence and popularity,
customizing these models using unauthorized data also brings serious privacy
and intellectual property issues. Existing methods introduce protective
perturbations based on adversarial attacks, which are applied to the
customization samples. In this systematization of knowledge, we present a
comprehensive survey of protective perturbation methods designed to prevent
unauthorized data usage in diffusion-based image generation. We establish the
threat model and categorize the downstream tasks relevant to these methods,
providing a detailed analysis of their designs. We also propose a completed
evaluation framework for these perturbation techniques, aiming to advance
research in this field.


http://arxiv.org/abs/2412.19037
CL-attack: Textual Backdoor Attacks via Cross-Lingual Triggers. (9%)
Jingyi Zheng; Tianyi Hu; Tianshuo Cong; Xinlei He
  Backdoor attacks significantly compromise the security of large language
models by triggering them to output specific and controlled content. Currently,
triggers for textual backdoor attacks fall into two categories: fixed-token
triggers and sentence-pattern triggers. However, the former are typically easy
to identify and filter, while the latter, such as syntax and style, do not
apply to all original samples and may lead to semantic shifts. In this paper,
inspired by cross-lingual (CL) prompts of LLMs in real-world scenarios, we
propose a higher-dimensional trigger method at the paragraph level, namely
CL-attack. CL-attack injects the backdoor by using texts with specific
structures that incorporate multiple languages, thereby offering greater
stealthiness and universality compared to existing backdoor attack techniques.
Extensive experiments on different tasks and model architectures demonstrate
that CL-attack can achieve nearly 100% attack success rate with a low poisoning
rate in both classification and generation tasks. We also empirically show that
the CL-attack is more robust against current major defense methods compared to
baseline backdoor attacks. Additionally, to mitigate CL-attack, we further
develop a new defense called TranslateDefense, which can partially mitigate the
impact of CL-attack.


http://arxiv.org/abs/2412.18706
SurvAttack: Black-Box Attack On Survival Models through Ontology-Informed EHR Perturbation. (99%)
Mohsen Nayebi Kerdabadi; Arya Hadizadeh Moghaddam; Bin Liu; Mei Liu; Zijun Yao
  Survival analysis (SA) models have been widely studied in mining electronic
health records (EHRs), particularly in forecasting the risk of critical
conditions for prioritizing high-risk patients. However, their vulnerability to
adversarial attacks is much less explored in the literature. Developing
black-box perturbation algorithms and evaluating their impact on
state-of-the-art survival models brings two benefits to medical applications.
First, it can effectively evaluate the robustness of models in pre-deployment
testing. Also, exploring how subtle perturbations would result in significantly
different outcomes can provide counterfactual insights into the clinical
interpretation of model prediction. In this work, we introduce SurvAttack, a
novel black-box adversarial attack framework leveraging subtle clinically
compatible, and semantically consistent perturbations on longitudinal EHRs to
degrade survival models' predictive performance. We specifically develop a
greedy algorithm to manipulate medical codes with various adversarial actions
throughout a patient's medical history. Then, these adversarial actions are
prioritized using a composite scoring strategy based on multi-aspect
perturbation quality, including saliency, perturbation stealthiness, and
clinical meaningfulness. The proposed adversarial EHR perturbation algorithm is
then used in an efficient SA-specific strategy to attack a survival model when
estimating the temporal ranking of survival urgency for patients. To
demonstrate the significance of our work, we conduct extensive experiments,
including baseline comparisons, explainability analysis, and case studies. The
experimental results affirm our research's effectiveness in illustrating the
vulnerabilities of patient survival models, model interpretation, and
ultimately contributing to healthcare quality.


http://arxiv.org/abs/2412.18718
Evaluating the Adversarial Robustness of Detection Transformers. (99%)
Amirhossein Nazeri; Chunheng Zhao; Pierluigi Pisu
  Robust object detection is critical for autonomous driving and mobile
robotics, where accurate detection of vehicles, pedestrians, and obstacles is
essential for ensuring safety. Despite the advancements in object detection
transformers (DETRs), their robustness against adversarial attacks remains
underexplored. This paper presents a comprehensive evaluation of DETR model and
its variants under both white-box and black-box adversarial attacks, using the
MS-COCO and KITTI datasets to cover general and autonomous driving scenarios.
We extend prominent white-box attack methods (FGSM, PGD, and CW) to assess DETR
vulnerability, demonstrating that DETR models are significantly susceptible to
adversarial attacks, similar to traditional CNN-based detectors. Our extensive
transferability analysis reveals high intra-network transferability among DETR
variants, but limited cross-network transferability to CNN-based models.
Additionally, we propose a novel untargeted attack designed specifically for
DETR, exploiting its intermediate loss functions to induce misclassification
with minimal perturbations. Visualizations of self-attention feature maps
provide insights into how adversarial attacks affect the internal
representations of DETR models. These findings reveal critical vulnerabilities
in detection transformers under standard adversarial attacks, emphasizing the
need for future research to enhance the robustness of transformer-based object
detectors in safety-critical applications.


http://arxiv.org/abs/2412.18196
Robustness-aware Automatic Prompt Optimization. (98%)
Zeru Shi; Zhenting Wang; Yongye Su; Weidi Luo; Fan Yang; Yongfeng Zhang
  The performance of Large Language Models (LLMs) is based on the quality of
the prompts and the semantic and structural integrity information of the input
data. However, current prompt generation methods primarily focus on generating
prompts for clean input data, often overlooking the impact of perturbed inputs
on prompt performance. To address this limitation, we propose BATprompt (By
Adversarial Training prompt), a novel method for prompt generation designed to
withstand input perturbations (such as typos in the input). Inspired by
adversarial training techniques, BATprompt demonstrates strong performance on a
variety of perturbed tasks through a two-step process: adversarial perturbation
and iterative optimization on unperturbed input via LLM. Unlike conventional
adversarial attack methods, BATprompt avoids reliance on real gradients or
model parameters. Instead, it leverages the advanced reasoning, language
understanding and self reflection capabilities of LLMs to simulate gradients,
guiding the generation of adversarial perturbations and optimizing prompt
performance. In our experiments, we evaluate BATprompt on multiple datasets
across both language understanding and generation tasks. The results indicate
that BATprompt outperforms existing prompt generation methods, delivering
superior robustness and performance under diverse perturbation scenarios.


http://arxiv.org/abs/2412.18770
Attack-in-the-Chain: Bootstrapping Large Language Models for Attacks Against Black-box Neural Ranking Models. (98%)
Yu-An Liu; Ruqing Zhang; Jiafeng Guo; Rijke Maarten de; Yixing Fan; Xueqi Cheng
  Neural ranking models (NRMs) have been shown to be highly effective in terms
of retrieval performance. Unfortunately, they have also displayed a higher
degree of sensitivity to attacks than previous generation models. To help
expose and address this lack of robustness, we introduce a novel ranking attack
framework named Attack-in-the-Chain, which tracks interactions between large
language models (LLMs) and NRMs based on chain-of-thought (CoT) prompting to
generate adversarial examples under black-box settings. Our approach starts by
identifying anchor documents with higher ranking positions than the target
document as nodes in the reasoning chain. We then dynamically assign the number
of perturbation words to each node and prompt LLMs to execute attacks. Finally,
we verify the attack performance of all nodes at each reasoning step and
proceed to generate the next reasoning step. Empirical results on two web
search benchmarks show the effectiveness of our method.


http://arxiv.org/abs/2412.18218
On the Effectiveness of Adversarial Training on Malware Classifiers. (96%)
Hamid Bostani; Jacopo Cortellazzi; Daniel Arp; Fabio Pierazzi; Veelasha Moonsamy; Lorenzo Cavallaro
  Adversarial Training (AT) has been widely applied to harden learning-based
classifiers against adversarial evasive attacks. However, its effectiveness in
identifying and strengthening vulnerable areas of the model's decision space
while maintaining high performance on clean data of malware classifiers remains
an under-explored area. In this context, the robustness that AT achieves has
often been assessed against unrealistic or weak adversarial attacks, which
negatively affect performance on clean data and are arguably no longer threats.
Previous work seems to suggest robustness is a task-dependent property of AT.
We instead argue it is a more complex problem that requires exploring AT and
the intertwined roles played by certain factors within data, feature
representations, classifiers, and robust optimization settings, as well as
proper evaluation factors, such as the realism of evasion attacks, to gain a
true sense of AT's effectiveness. In our paper, we address this gap by
systematically exploring the role such factors have in hardening malware
classifiers through AT. Contrary to recent prior work, a key observation of our
research and extensive experiments confirm the hypotheses that all such factors
influence the actual effectiveness of AT, as demonstrated by the varying
degrees of success from our empirical analysis. We identify five evaluation
pitfalls that affect state-of-the-art studies and summarize our insights in ten
takeaways to draw promising research directions toward better understanding the
factors' settings under which adversarial training works at best.


http://arxiv.org/abs/2412.18262
Efficient Contrastive Explanations on Demand. (82%)
Yacine Izza; Joao Marques-Silva
  Recent work revealed a tight connection between adversarial robustness and
restricted forms of symbolic explanations, namely distance-based (formal)
explanations. This connection is significant because it represents a first step
towards making the computation of symbolic explanations as efficient as
deciding the existence of adversarial examples, especially for highly complex
machine learning (ML) models. However, a major performance bottleneck remains,
because of the very large number of features that ML models may possess, in
particular for deep neural networks. This paper proposes novel algorithms to
compute the so-called contrastive explanations for ML models with a large
number of features, by leveraging on adversarial robustness. Furthermore, the
paper also proposes novel algorithms for listing explanations and finding
smallest contrastive explanations. The experimental results demonstrate the
performance gains achieved by the novel algorithms proposed in this paper.


http://arxiv.org/abs/2412.18507
An Empirical Analysis of Federated Learning Models Subject to Label-Flipping Adversarial Attack. (73%)
Kunal Bhatnagar; Sagana Chattanathan; Angela Dang; Bhargav Eranki; Ronnit Rana; Charan Sridhar; Siddharth Vedam; Angie Yao; Mark Stamp
  In this paper, we empirically analyze adversarial attacks on selected
federated learning models. The specific learning models considered are
Multinominal Logistic Regression (MLR), Support Vector Classifier (SVC),
Multilayer Perceptron (MLP), Convolution Neural Network (CNN), %Recurrent
Neural Network (RNN), Random Forest, XGBoost, and Long Short-Term Memory
(LSTM). For each model, we simulate label-flipping attacks, experimenting
extensively with 10 federated clients and 100 federated clients. We vary the
percentage of adversarial clients from 10% to 100% and, simultaneously, the
percentage of labels flipped by each adversarial client is also varied from 10%
to 100%. Among other results, we find that models differ in their inherent
robustness to the two vectors in our label-flipping attack, i.e., the
percentage of adversarial clients, and the percentage of labels flipped by each
adversarial client. We discuss the potential practical implications of our
results.


http://arxiv.org/abs/2412.18370
Unveiling the Threat of Fraud Gangs to Graph Neural Networks: Multi-Target Graph Injection Attacks Against GNN-Based Fraud Detectors. (41%)
Jinhyeok Choi; Heehyeon Kim; Joyce Jiyoung Whang
  Graph neural networks (GNNs) have emerged as an effective tool for fraud
detection, identifying fraudulent users, and uncovering malicious behaviors.
However, attacks against GNN-based fraud detectors and their risks have rarely
been studied, thereby leaving potential threats unaddressed. Recent findings
suggest that frauds are increasingly organized as gangs or groups. In this
work, we design attack scenarios where fraud gangs aim to make their fraud
nodes misclassified as benign by camouflaging their illicit activities in
collusion. Based on these scenarios, we study adversarial attacks against
GNN-based fraud detectors by simulating attacks of fraud gangs in three
real-world fraud cases: spam reviews, fake news, and medical insurance frauds.
We define these attacks as multi-target graph injection attacks and propose
MonTi, a transformer-based Multi-target one-Time graph injection attack model.
MonTi simultaneously generates attributes and edges of all attack nodes with a
transformer encoder, capturing interdependencies between attributes and edges
more effectively than most existing graph injection attack methods that
generate these elements sequentially. Additionally, MonTi adaptively allocates
the degree budget for each attack node to explore diverse injection structures
involving target, candidate, and attack nodes, unlike existing methods that fix
the degree budget across all attack nodes. Experiments show that MonTi
outperforms the state-of-the-art graph injection attack methods on five
real-world graphs.


http://arxiv.org/abs/2412.18171
Token Highlighter: Inspecting and Mitigating Jailbreak Prompts for Large Language Models. (38%)
Xiaomeng Hu; Pin-Yu Chen; Tsung-Yi Ho
  Large Language Models (LLMs) are increasingly being integrated into services
such as ChatGPT to provide responses to user queries. To mitigate potential
harm and prevent misuse, there have been concerted efforts to align the LLMs
with human values and legal compliance by incorporating various techniques,
such as Reinforcement Learning from Human Feedback (RLHF), into the training of
the LLMs. However, recent research has exposed that even aligned LLMs are
susceptible to adversarial manipulations known as Jailbreak Attacks. To address
this challenge, this paper proposes a method called Token Highlighter to
inspect and mitigate the potential jailbreak threats in the user query. Token
Highlighter introduced a concept called Affirmation Loss to measure the LLM's
willingness to answer the user query. It then uses the gradient of Affirmation
Loss for each token in the user query to locate the jailbreak-critical tokens.
Further, Token Highlighter exploits our proposed Soft Removal technique to
mitigate the jailbreak effects of critical tokens via shrinking their token
embeddings. Experimental results on two aligned LLMs (LLaMA-2 and Vicuna-V1.5)
demonstrate that the proposed method can effectively defend against a variety
of Jailbreak Attacks while maintaining competent performance on benign
questions of the AlpacaEval benchmark. In addition, Token Highlighter is a
cost-effective and interpretable defense because it only needs to query the
protected LLM once to compute the Affirmation Loss and can highlight the
critical tokens upon refusal.


http://arxiv.org/abs/2412.18365
Hypergraph Attacks via Injecting Homogeneous Nodes into Elite Hyperedges. (22%)
Meixia He; Peican Zhu; Keke Tang; Yangming Guo
  Recent studies have shown that Hypergraph Neural Networks (HGNNs) are
vulnerable to adversarial attacks. Existing approaches focus on hypergraph
modification attacks guided by gradients, overlooking node spanning in the
hypergraph and the group identity of hyperedges, thereby resulting in limited
attack performance and detectable attacks. In this manuscript, we present a
novel framework, i.e., Hypergraph Attacks via Injecting Homogeneous Nodes into
Elite Hyperedges (IE-Attack), to tackle these challenges. Initially, utilizing
the node spanning in the hypergraph, we propose the elite hyperedges sampler to
identify hyperedges to be injected. Subsequently, a node generator utilizing
Kernel Density Estimation (KDE) is proposed to generate the homogeneous node
with the group identity of hyperedges. Finally, by injecting the homogeneous
node into elite hyperedges, IE-Attack improves the attack performance and
enhances the imperceptibility of attacks. Extensive experiments are conducted
on five authentic datasets to validate the effectiveness of IE-Attack and the
corresponding superiority to state-of-the-art methods.


http://arxiv.org/abs/2412.18409
Re-assessing ImageNet: How aligned is its single-label assumption with its multi-label nature? (4%)
Esla Timothy Anzaku; Seyed Amir Mousavi; Messem Arnout Van; Neve Wesley De
  ImageNet, an influential dataset in computer vision, is traditionally
evaluated using single-label classification, which assumes that an image can be
adequately described by a single concept or label. However, this approach may
not fully capture the complex semantics within the images available in
ImageNet, potentially hindering the development of models that effectively
learn these intricacies. This study critically examines the prevalent
single-label benchmarking approach and advocates for a shift to multi-label
benchmarking for ImageNet. This shift would enable a more comprehensive
assessment of the capabilities of deep neural network (DNN) models. We analyze
the effectiveness of pre-trained state-of-the-art DNNs on ImageNet and one of
its variants, ImageNetV2. Studies in the literature have reported unexpected
accuracy drops of 11% to 14% on ImageNetV2. Our findings show that these
reported declines are largely attributable to a characteristic of the dataset
that has not received sufficient attention -- the proportion of images with
multiple labels. Taking this characteristic into account, the results of our
experiments provide evidence that there is no substantial degradation in
effectiveness on ImageNetV2. Furthermore, we acknowledge that ImageNet
pre-trained models exhibit some capability at capturing the multi-label nature
of the dataset even though they were trained under the single-label assumption.
Consequently, we propose a new evaluation approach to augment existing
approaches that assess this capability. Our findings highlight the importance
of considering the multi-label nature of the ImageNet dataset during
benchmarking. Failing to do so could lead to incorrect conclusions regarding
the effectiveness of DNNs and divert research efforts from addressing other
substantial challenges related to the reliability and robustness of these
models.


http://arxiv.org/abs/2412.17544
Retention Score: Quantifying Jailbreak Risks for Vision Language Models. (99%)
Zaitang Li; Pin-Yu Chen; Tsung-Yi Ho
  The emergence of Vision-Language Models (VLMs) is a significant advancement
in integrating computer vision with Large Language Models (LLMs) to enhance
multi-modal machine learning capabilities. However, this progress has also made
VLMs vulnerable to sophisticated adversarial attacks, raising concerns about
their reliability. The objective of this paper is to assess the resilience of
VLMs against jailbreak attacks that can compromise model safety compliance and
result in harmful outputs. To evaluate a VLM's ability to maintain its
robustness against adversarial input perturbations, we propose a novel metric
called the \textbf{Retention Score}. Retention Score is a multi-modal
evaluation metric that includes Retention-I and Retention-T scores for
quantifying jailbreak risks in visual and textual components of VLMs. Our
process involves generating synthetic image-text pairs using a conditional
diffusion model. These pairs are then predicted for toxicity score by a VLM
alongside a toxicity judgment classifier. By calculating the margin in toxicity
scores, we can quantify the robustness of the VLM in an attack-agnostic manner.
Our work has four main contributions. First, we prove that Retention Score can
serve as a certified robustness metric. Second, we demonstrate that most VLMs
with visual components are less robust against jailbreak attacks than the
corresponding plain VLMs. Additionally, we evaluate black-box VLM APIs and find
that the security settings in Google Gemini significantly affect the score and
robustness. Moreover, the robustness of GPT4V is similar to the medium settings
of Gemini. Finally, our approach offers a time-efficient alternative to
existing adversarial attack methods and provides consistent model robustness
rankings when evaluated on VLMs including MiniGPT-4, InstructBLIP, and LLaVA.


http://arxiv.org/abs/2412.17614
Emerging Security Challenges of Large Language Models. (73%)
Herve Debar; Sven Dietrich; Pavel Laskov; Emil C. Lupu; Eirini Ntoutsi
  Large language models (LLMs) have achieved record adoption in a short period
of time across many different sectors including high importance areas such as
education [4] and healthcare [23]. LLMs are open-ended models trained on
diverse data without being tailored for specific downstream tasks, enabling
broad applicability across various domains. They are commonly used for text
generation, but also widely used to assist with code generation [3], and even
analysis of security information, as Microsoft Security Copilot demonstrates
[18]. Traditional Machine Learning (ML) models are vulnerable to adversarial
attacks [9]. So the concerns on the potential security implications of such
wide scale adoption of LLMs have led to the creation of this working group on
the security of LLMs. During the Dagstuhl seminar on "Network Attack Detection
and Defense - AI-Powered Threats and Responses", the working group discussions
focused on the vulnerability of LLMs to adversarial attacks, rather than their
potential use in generating malware or enabling cyberattacks. Although we note
the potential threat represented by the latter, the role of the LLMs in such
uses is mostly as an accelerator for development, similar to what it is in
benign use. To make the analysis more specific, the working group employed
ChatGPT as a concrete example of an LLM and addressed the following points,
which also form the structure of this report: 1. How do LLMs differ in
vulnerabilities from traditional ML models? 2. What are the attack objectives
in LLMs? 3. How complex it is to assess the risks posed by the vulnerabilities
of LLMs? 4. What is the supply chain in LLMs, how data flow in and out of
systems and what are the security implications? We conclude with an overview of
open challenges and outlook.


http://arxiv.org/abs/2412.17888
Stability Bounds for the Unfolded Forward-Backward Algorithm. (38%)
Emilie Chouzenoux; Valle Cecile Della; Jean-Christophe Pesquet
  We consider a neural network architecture designed to solve inverse problems
where the degradation operator is linear and known. This architecture is
constructed by unrolling a forward-backward algorithm derived from the
minimization of an objective function that combines a data-fidelity term, a
Tikhonov-type regularization term, and a potentially nonsmooth convex penalty.
The robustness of this inversion method to input perturbations is analyzed
theoretically. Ensuring robustness complies with the principles of inverse
problem theory, as it ensures both the continuity of the inversion method and
the resilience to small noise - a critical property given the known
vulnerability of deep neural networks to adversarial perturbations. A key
novelty of our work lies in examining the robustness of the proposed network to
perturbations in its bias, which represents the observed data in the inverse
problem. Additionally, we provide numerical illustrations of the analytical
Lipschitz bounds derived in our analysis.


http://arxiv.org/abs/2412.17531
Double Landmines: Invisible Textual Backdoor Attacks based on Dual-Trigger. (15%)
Yang Hou; Qiuling Yue; Lujia Chai; Guozhao Liao; Wenbao Han; Wei Ou
  At present, all textual backdoor attack methods are based on single triggers:
for example, inserting specific content into the text to activate the backdoor;
or changing the abstract text features. The former is easier to be identified
by existing defense strategies due to its obvious characteristics; the latter,
although improved in invisibility, has certain shortcomings in terms of attack
performance, construction of poisoned datasets, and selection of the final
poisoning rate. On this basis, this paper innovatively proposes a Dual-Trigger
backdoor attack based on syntax and mood, and optimizes the construction of the
poisoned dataset and the selection strategy of the final poisoning rate. A
large number of experimental results show that this method significantly
outperforms the previous methods based on abstract features in attack
performance, and achieves comparable attack performance (almost 100% attack
success rate) with the insertion-based method. In addition, the two trigger
mechanisms included in this method can be activated independently in the
application phase of the model, which not only improves the flexibility of the
trigger style, but also enhances its robustness against defense strategies.
These results profoundly reveal that textual backdoor attacks are extremely
harmful and provide a new perspective for security protection in this field.


http://arxiv.org/abs/2412.18123
AEIOU: A Unified Defense Framework against NSFW Prompts in Text-to-Image Models. (2%)
Yiming Wang; Jiahao Chen; Qingming Li; Xing Yang; Shouling Ji
  As text-to-image (T2I) models continue to advance and gain widespread
adoption, their associated safety issues are becoming increasingly prominent.
Malicious users often exploit these models to generate Not-Safe-for-Work (NSFW)
images using harmful or adversarial prompts, highlighting the critical need for
robust safeguards to ensure the integrity and compliance of model outputs.
Current internal safeguards frequently degrade image quality, while external
detection methods often suffer from low accuracy and inefficiency.
  In this paper, we introduce AEIOU, a defense framework that is Adaptable,
Efficient, Interpretable, Optimizable, and Unified against NSFW prompts in T2I
models. AEIOU extracts NSFW features from the hidden states of the model's text
encoder, utilizing the separable nature of these features to detect NSFW
prompts. The detection process is efficient, requiring minimal inference time.
AEIOU also offers real-time interpretation of results and supports optimization
through data augmentation techniques. The framework is versatile, accommodating
various T2I architectures. Our extensive experiments show that AEIOU
significantly outperforms both commercial and open-source moderation tools,
achieving over 95% accuracy across all datasets and improving efficiency by at
least tenfold. It effectively counters adaptive attacks and excels in few-shot
and multi-label scenarios.


http://arxiv.org/abs/2412.17740
Sensitivity Curve Maximization: Attacking Robust Aggregators in Distributed Learning. (1%)
Christian A. Schroth; Stefan Vlaski; Abdelhak M. Zoubir
  In distributed learning agents aim at collaboratively solving a global
learning problem. It becomes more and more likely that individual agents are
malicious or faulty with an increasing size of the network. This leads to a
degeneration or complete breakdown of the learning process. Classical
aggregation schemes are prone to breakdown at small contamination rates,
therefore robust aggregation schemes are sought for. While robust aggregation
schemes can generally tolerate larger contamination rates, many have been shown
to be susceptible to carefully crafted malicious attacks. In this work, we show
how the sensitivity curve (SC), a classical tool from robust statistics, can be
used to systematically derive optimal attack patterns against arbitrary robust
aggregators, in most cases rendering them ineffective. We show the
effectiveness of the proposed attack in multiple simulations.


http://arxiv.org/abs/2412.17522
DiffusionAttacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreak. (1%)
Hao Wang; Hao Li; Junda Zhu; Xinyuan Wang; Chengwei Pan; MinLie Huang; Lei Sha
  Large Language Models (LLMs) are susceptible to generating harmful content
when prompted with carefully crafted inputs, a vulnerability known as LLM
jailbreaking. As LLMs become more powerful, studying jailbreak methods is
critical to enhancing security and aligning models with human values.
Traditionally, jailbreak techniques have relied on suffix addition or prompt
templates, but these methods suffer from limited attack diversity. This paper
introduces DiffusionAttacker, an end-to-end generative approach for jailbreak
rewriting inspired by diffusion models. Our method employs a
sequence-to-sequence (seq2seq) text diffusion model as a generator,
conditioning on the original prompt and guiding the denoising process with a
novel attack loss. Unlike previous approaches that use autoregressive LLMs to
generate jailbreak prompts, which limit the modification of already generated
tokens and restrict the rewriting space, DiffusionAttacker utilizes a seq2seq
diffusion model, allowing more flexible token modifications. This approach
preserves the semantic content of the original prompt while producing harmful
content. Additionally, we leverage the Gumbel-Softmax technique to make the
sampling process from the diffusion model's output distribution differentiable,
eliminating the need for iterative token search. Extensive experiments on
Advbench and Harmbench demonstrate that DiffusionAttacker outperforms previous
methods across various evaluation metrics, including attack success rate (ASR),
fluency, and diversity.


http://arxiv.org/abs/2412.17038
ErasableMask: A Robust and Erasable Privacy Protection Scheme against Black-box Face Recognition Models. (99%)
Sipeng Shen; Yunming Zhang; Dengpan Ye; Xiuwen Shi; Long Tang; Haoran Duan; Ziyi Liu
  While face recognition (FR) models have brought remarkable convenience in
face verification and identification, they also pose substantial privacy risks
to the public. Existing facial privacy protection schemes usually adopt
adversarial examples to disrupt face verification of FR models. However, these
schemes often suffer from weak transferability against black-box FR models and
permanently damage the identifiable information that cannot fulfill the
requirements of authorized operations such as forensics and authentication. To
address these limitations, we propose ErasableMask, a robust and erasable
privacy protection scheme against black-box FR models. Specifically, via
rethinking the inherent relationship between surrogate FR models, ErasableMask
introduces a novel meta-auxiliary attack, which boosts black-box
transferability by learning more general features in a stable and balancing
optimization strategy. It also offers a perturbation erasion mechanism that
supports the erasion of semantic perturbations in protected face without
degrading image quality. To further improve performance, ErasableMask employs a
curriculum learning strategy to mitigate optimization conflicts between
adversarial attack and perturbation erasion. Extensive experiments on the
CelebA-HQ and FFHQ datasets demonstrate that ErasableMask achieves the
state-of-the-art performance in transferability, achieving over 72% confidence
on average in commercial FR systems. Moreover, ErasableMask also exhibits
outstanding perturbation erasion performance, achieving over 90% erasion
success rate.


http://arxiv.org/abs/2412.16955
NumbOD: A Spatial-Frequency Fusion Attack Against Object Detectors. (99%)
Ziqi Zhou; Bowen Li; Yufei Song; Zhifei Yu; Shengshan Hu; Wei Wan; Leo Yu Zhang; Dezhong Yao; Hai Jin
  With the advancement of deep learning, object detectors (ODs) with various
architectures have achieved significant success in complex scenarios like
autonomous driving. Previous adversarial attacks against ODs have been focused
on designing customized attacks targeting their specific structures (e.g., NMS
and RPN), yielding some results but simultaneously constraining their
scalability. Moreover, most efforts against ODs stem from image-level attacks
originally designed for classification tasks, resulting in redundant
computations and disturbances in object-irrelevant areas (e.g., background).
Consequently, how to design a model-agnostic efficient attack to
comprehensively evaluate the vulnerabilities of ODs remains challenging and
unresolved. In this paper, we propose NumbOD, a brand-new spatial-frequency
fusion attack against various ODs, aimed at disrupting object detection within
images. We directly leverage the features output by the OD without relying on
its internal structures to craft adversarial examples. Specifically, we first
design a dual-track attack target selection strategy to select high-quality
bounding boxes from OD outputs for targeting. Subsequently, we employ
directional perturbations to shift and compress predicted boxes and change
classification results to deceive ODs. Additionally, we focus on manipulating
the high-frequency components of images to confuse ODs' attention on critical
objects, thereby enhancing the attack efficiency. Our extensive experiments on
nine ODs and two datasets show that NumbOD achieves powerful attack performance
and high stealthiness.


http://arxiv.org/abs/2412.16958
Breaking Barriers in Physical-World Adversarial Examples: Improving Robustness and Transferability via Robust Feature. (99%)
Yichen Wang; Yuxuan Chou; Ziqi Zhou; Hangtao Zhang; Wei Wan; Shengshan Hu; Minghui Li
  As deep neural networks (DNNs) are widely applied in the physical world, many
researches are focusing on physical-world adversarial examples (PAEs), which
introduce perturbations to inputs and cause the model's incorrect outputs.
However, existing PAEs face two challenges: unsatisfactory attack performance
(i.e., poor transferability and insufficient robustness to environment
conditions), and difficulty in balancing attack effectiveness with
stealthiness, where better attack effectiveness often makes PAEs more
perceptible.
  In this paper, we explore a novel perturbation-based method to overcome the
challenges. For the first challenge, we introduce a strategy Deceptive RF
injection based on robust features (RFs) that are predictive, robust to
perturbations, and consistent across different models. Specifically, it
improves the transferability and robustness of PAEs by covering RFs of other
classes onto the predictive features in clean images. For the second challenge,
we introduce another strategy Adversarial Semantic Pattern Minimization, which
removes most perturbations and retains only essential adversarial patterns in
AEsBased on the two strategies, we design our method Robust Feature Coverage
Attack (RFCoA), comprising Robust Feature Disentanglement and Adversarial
Feature Fusion. In the first stage, we extract target class RFs in feature
space. In the second stage, we use attention-based feature fusion to overlay
these RFs onto predictive features of clean images and remove unnecessary
perturbations. Experiments show our method's superior transferability,
robustness, and stealthiness compared to existing state-of-the-art methods.
Additionally, our method's effectiveness can extend to Large Vision-Language
Models (LVLMs), indicating its potential applicability to more complex tasks.


http://arxiv.org/abs/2412.16893
Preventing Non-intrusive Load Monitoring Privacy Invasion: A Precise Adversarial Attack Scheme for Networked Smart Meters. (98%)
Jialing He; Jiacheng Wang; Ning Wang; Shangwei Guo; Liehuang Zhu; Dusit Niyato; Tao Xiang
  Smart grid, through networked smart meters employing the non-intrusive load
monitoring (NILM) technique, can considerably discern the usage patterns of
residential appliances. However, this technique also incurs privacy leakage. To
address this issue, we propose an innovative scheme based on adversarial attack
in this paper. The scheme effectively prevents NILM models from violating
appliance-level privacy, while also ensuring accurate billing calculation for
users. To achieve this objective, we overcome two primary challenges. First, as
NILM models fall under the category of time-series regression models, direct
application of traditional adversarial attacks designed for classification
tasks is not feasible. To tackle this issue, we formulate a novel adversarial
attack problem tailored specifically for NILM and providing a theoretical
foundation for utilizing the Jacobian of the NILM model to generate
imperceptible perturbations. Leveraging the Jacobian, our scheme can produce
perturbations, which effectively misleads the signal prediction of NILM models
to safeguard users' appliance-level privacy. The second challenge pertains to
fundamental utility requirements, where existing adversarial attack schemes
struggle to achieve accurate billing calculation for users. To handle this
problem, we introduce an additional constraint, mandating that the sum of added
perturbations within a billing period must be precisely zero. Experimental
validation on real-world power datasets REDD and UK-DALE demonstrates the
efficacy of our proposed solutions, which can significantly amplify the
discrepancy between the output of the targeted NILM model and the actual power
signal of appliances, and enable accurate billing at the same time.
Additionally, our solutions exhibit transferability, making the generated
perturbation signal from one target model applicable to other diverse NILM
models.


http://arxiv.org/abs/2412.16905
A Backdoor Attack Scheme with Invisible Triggers Based on Model Architecture Modification. (45%)
Yuan Ma; Xu Ma; Jiankang Wei; Jinmeng Tang; Xiaoyu Zhang; Yilun Lyu; Kehao Chen; Jingtong Huang
  Machine learning systems are vulnerable to backdoor attacks, where attackers
manipulate model behavior through data tampering or architectural
modifications. Traditional backdoor attacks involve injecting malicious samples
with specific triggers into the training data, causing the model to produce
targeted incorrect outputs in the presence of the corresponding triggers. More
sophisticated attacks modify the model's architecture directly, embedding
backdoors that are harder to detect as they evade traditional data-based
detection methods. However, the drawback of the architectural modification
based backdoor attacks is that the trigger must be visible in order to activate
the backdoor. To further strengthen the invisibility of the backdoor attacks, a
novel backdoor attack method is presented in the paper. To be more specific,
this method embeds the backdoor within the model's architecture and has the
capability to generate inconspicuous and stealthy triggers. The attack is
implemented by modifying pre-trained models, which are then redistributed,
thereby posing a potential threat to unsuspecting users. Comprehensive
experiments conducted on standard computer vision benchmarks validate the
effectiveness of this attack and highlight the stealthiness of its triggers,
which remain undetectable through both manual visual inspection and advanced
detection tools.


http://arxiv.org/abs/2412.17213
Attack by Yourself: Effective and Unnoticeable Multi-Category Graph Backdoor Attacks with Subgraph Triggers Pool. (22%)
Jiangtong Li; Dungy Liu; Dawei Cheng; Changchun Jiang
  \textbf{G}raph \textbf{N}eural \textbf{N}etworks~(GNNs) have achieved
significant success in various real-world applications, including social
networks, finance systems, and traffic management. Recent researches highlight
their vulnerability to backdoor attacks in node classification, where GNNs
trained on a poisoned graph misclassify a test node only when specific triggers
are attached. These studies typically focus on single attack categories and use
adaptive trigger generators to create node-specific triggers. However, adaptive
trigger generators typically have a simple structure, limited parameters, and
lack category-aware graph knowledge, which makes them struggle to handle
backdoor attacks across multiple categories as the number of target categories
increases. We address this gap by proposing a novel approach for
\textbf{E}ffective and \textbf{U}nnoticeable
\textbf{M}ulti-\textbf{C}ategory~(EUMC) graph backdoor attacks, leveraging
subgraph from the attacked graph as category-aware triggers to precisely
control the target category. To ensure the effectiveness of our method, we
construct a \textbf{M}ulti-\textbf{C}ategory \textbf{S}ubgraph
\textbf{T}riggers \textbf{P}ool~(MC-STP) using the subgraphs of the attacked
graph as triggers. We then exploit the attachment probability shifts of each
subgraph trigger as category-aware priors for target category determination.
Moreover, we develop a ``select then attach'' strategy that connects suitable
category-aware trigger to attacked nodes for unnoticeability. Extensive
experiments across different real-world datasets confirm the efficacy of our
method in conducting multi-category graph backdoor attacks on various GNN
models and defense strategies.


http://arxiv.org/abs/2412.17034
Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Models. (22%)
Lang Gao; Xiangliang Zhang; Preslav Nakov; Xiuying Chen
  Jailbreaking in Large Language Models (LLMs) is a major security concern as
it can deceive LLMs to generate harmful text. Yet, there is still insufficient
understanding of how jailbreaking works, which makes it hard to develop
effective defense strategies. We aim to shed more light into this issue: we
conduct a detailed large-scale analysis of seven different jailbreak methods
and find that these disagreements stem from insufficient observation samples.
In particular, we introduce \textit{safety boundary}, and we find that
jailbreaks shift harmful activations outside that safety boundary, where LLMs
are less sensitive to harmful information. We also find that the low and the
middle layers are critical in such shifts, while deeper layers have less
impact. Leveraging on these insights, we propose a novel defense called
\textbf{Activation Boundary Defense} (ABD), which adaptively constrains the
activations within the safety boundary. We further use Bayesian optimization to
selectively apply the defense method to the low and the middle layers. Our
experiments on several benchmarks show that ABD achieves an average DSR of over
98\% against various forms of jailbreak attacks, with less than 2\% impact on
the model's general capabilities.


http://arxiv.org/abs/2412.17011
Robustness of Large Language Models Against Adversarial Attacks. (13%)
Yiyi Tao; Yixian Shen; Hang Zhang; Yanxin Shen; Lun Wang; Chuanqi Shi; Shaoshuai Du
  The increasing deployment of Large Language Models (LLMs) in various
applications necessitates a rigorous evaluation of their robustness against
adversarial attacks. In this paper, we present a comprehensive study on the
robustness of GPT LLM family. We employ two distinct evaluation methods to
assess their resilience. The first method introduce character-level text attack
in input prompts, testing the models on three sentiment classification
datasets: StanfordNLP/IMDB, Yelp Reviews, and SST-2. The second method involves
using jailbreak prompts to challenge the safety mechanisms of the LLMs. Our
experiments reveal significant variations in the robustness of these models,
demonstrating their varying degrees of vulnerability to both character-level
and semantic-level adversarial attacks. These findings underscore the necessity
for improved adversarial training and enhanced safety mechanisms to bolster the
robustness of LLMs.


http://arxiv.org/abs/2412.16651
PB-UAP: Hybrid Universal Adversarial Attack For Image Segmentation. (99%)
Yufei Song; Ziqi Zhou; Minghui Li; Xianlong Wang; Menghao Deng; Wei Wan; Shengshan Hu; Leo Yu Zhang
  With the rapid advancement of deep learning, the model robustness has become
a significant research hotspot, \ie, adversarial attacks on deep neural
networks. Existing works primarily focus on image classification tasks, aiming
to alter the model's predicted labels. Due to the output complexity and deeper
network architectures, research on adversarial examples for segmentation models
is still limited, particularly for universal adversarial perturbations. In this
paper, we propose a novel universal adversarial attack method designed for
segmentation models, which includes dual feature separation and low-frequency
scattering modules. The two modules guide the training of adversarial examples
in the pixel and frequency space, respectively. Experiments demonstrate that
our method achieves high attack success rates surpassing the state-of-the-art
methods, and exhibits strong transferability across different models.


http://arxiv.org/abs/2412.16662
Adversarial Attack Against Images Classification based on Generative Adversarial Networks. (98%)
Yahe Yang
  Adversarial attacks on image classification systems have always been an
important problem in the field of machine learning, and generative adversarial
networks (GANs), as popular models in the field of image generation, have been
widely used in various novel scenarios due to their powerful generative
capabilities. However, with the popularity of generative adversarial networks,
the misuse of fake image technology has raised a series of security problems,
such as malicious tampering with other people's photos and videos, and invasion
of personal privacy. Inspired by the generative adversarial networks, this work
proposes a novel adversarial attack method, aiming to gain insight into the
weaknesses of the image classification system and improve its anti-attack
ability. Specifically, the generative adversarial networks are used to generate
adversarial samples with small perturbations but enough to affect the
decision-making of the classifier, and the adversarial samples are generated
through the adversarial learning of the training generator and the classifier.
From extensive experiment analysis, we evaluate the effectiveness of the method
on a classical image classification dataset, and the results show that our
model successfully deceives a variety of advanced classifiers while maintaining
the naturalness of adversarial samples.


http://arxiv.org/abs/2412.16720
OpenAI o1 System Card. (81%)
OpenAI; :; Aaron Jaech; Adam Kalai; Adam Lerer; Adam Richardson; Ahmed El-Kishky; Aiden Low; Alec Helyar; Aleksander Madry; Alex Beutel; Alex Carney; Alex Iftimie; Alex Karpenko; Alex Tachard Passos; Alexander Neitz; Alexander Prokofiev; Alexander Wei; Allison Tam; Ally Bennett; Ananya Kumar; Andre Saraiva; Andrea Vallone; Andrew Duberstein; Andrew Kondrich; Andrey Mishchenko; Andy Applebaum; Angela Jiang; Ashvin Nair; Barret Zoph; Behrooz Ghorbani; Ben Rossen; Benjamin Sokolowsky; Boaz Barak; Bob McGrew; Borys Minaiev; Botao Hao; Bowen Baker; Brandon Houghton; Brandon McKinzie; Brydon Eastman; Camillo Lugaresi; Cary Bassin; Cary Hudson; Chak Ming Li; Bourcy Charles de; Chelsea Voss; Chen Shen; Chong Zhang; Chris Koch; Chris Orsinger; Christopher Hesse; Claudia Fischer; Clive Chan; Dan Roberts; Daniel Kappler; Daniel Levy; Daniel Selsam; David Dohan; David Farhi; David Mely; David Robinson; Dimitris Tsipras; Doug Li; Dragos Oprica; Eben Freeman; Eddie Zhang; Edmund Wong; Elizabeth Proehl; Enoch Cheung; Eric Mitchell; Eric Wallace; Erik Ritter; Evan Mays; Fan Wang; Felipe Petroski Such; Filippo Raso; Florencia Leoni; Foivos Tsimpourlas; Francis Song; Lohmann Fred von; Freddie Sulit; Geoff Salmon; Giambattista Parascandolo; Gildas Chabot; Grace Zhao; Greg Brockman; Guillaume Leclerc; Hadi Salman; Haiming Bao; Hao Sheng; Hart Andrin; Hessam Bagherinezhad; Hongyu Ren; Hunter Lightman; Hyung Won Chung; Ian Kivlichan; Ian O'Connell; Ian Osband; Ignasi Clavera Gilaberte; Ilge Akkaya; Ilya Kostrikov; Ilya Sutskever; Irina Kofman; Jakub Pachocki; James Lennon; Jason Wei; Jean Harb; Jerry Twore; Jiacheng Feng; Jiahui Yu; Jiayi Weng; Jie Tang; Jieqi Yu; Joaquin Quiñonero Candela; Joe Palermo; Joel Parish; Johannes Heidecke; John Hallman; John Rizzo; Jonathan Gordon; Jonathan Uesato; Jonathan Uesato; Jonathan Ward; Joost Huizinga; Julie Wang; Kai Chen; Kai Xiao; Karan Singhal; Karina Nguyen; Karl Cobbe; Katy Shi; Kayla Wood; Kendra Rimbach; Keren Gu-Lemberg; Keren GuLemberg; Kevin Liu; Kevin Lu; Kevin Stone; Kevin Yu; Lama Ahmad; Lauren Yang; Leo Liu; Leon Maksin; Leyton Ho; Liam Fedus; Lilian Weng; Linden Li; Lindsay McCallum; Lindsey Held; Lorenz Kuhn; Lukas Kondraciuk; Lukasz Kaiser; Luke Metz; Madelaine Boyd; Maja Trebacz; Manas Joglekar; Mark Chen; Marko Tintor; Mason Meyer; Matt Jones; Matt Kaufer; Max Schwarzer; Meghan Shah; Mehmet Yatbaz; Melody Guan; Mengyuan Xu; Mengyuan Yan; Mia Glaese; Mianna Chen; Mianna Chen; Michael Lampe; Michael Malek; Michele Wang; Michelle Fradin; Mike McClay; Mikhail Pavlov; Miles Wang; Mingxuan Wang; Mira Murati; Mo Bavarian; Mostafa Rohaninejad; Nat McAleese; Neil Chowdhury; Neil Chowdhury; Nick Ryder; Nikolas Tezak; Noam Brown; Ofir Nachum; Oleg Boiko; Oleg Murk; Olivia Watkins; Patrick Chao; Paul Ashbourne; Pavel Izmailov; Peter Zhokhov; Rachel Dias; Rahul Arora; Randall Lin; Rapha Gontijo Lopes; Raz Gaon; Reah Miyara; Reimar Leike; Renny Hwang; Rhythm Garg; Robin Brown; Roshan James; Rui Shu; Ryan Cheu; Ryan Greene; Saachi Jain; Sam Altman; Sam Toizer; Sam Toyer; Samuel Miserendino; Sandhini Agarwal; Santiago Hernandez; Sasha Baker; Scott McKinney; Scottie Yan; Shengjia Zhao; Shengli Hu; Shibani Santurkar; Shraman Ray Chaudhuri; Shuyuan Zhang; Siyuan Fu; Spencer Papay; Steph Lin; Suchir Balaji; Suvansh Sanjeev; Szymon Sidor; Tal Broda; Aidan Clark; Tao Wang; Taylor Gordon; Ted Sanders; Tejal Patwardhan; Thibault Sottiaux; Thomas Degry; Thomas Dimson; Tianhao Zheng; Timur Garipov; Tom Stasi; Trapit Bansal; Trevor Creech; Troy Peterson; Tyna Eloundou; Valerie Qi; Vineet Kosaraju; Vinnie Monaco; Vitchyr Pong; Vlad Fomenko; Weiyi Zheng; Wenda Zhou; Wes McCabe; Wojciech Zaremba; Yann Dubois; Yinghai Lu; Yining Chen; Young Cha; Yu Bai; Yuchen He; Yuchen Zhang; Yunyun Wang; Zheng Shao; Zhuohan Li
  The o1 model series is trained with large-scale reinforcement learning to
reason using chain of thought. These advanced reasoning capabilities provide
new avenues for improving the safety and robustness of our models. In
particular, our models can reason about our safety policies in context when
responding to potentially unsafe prompts, through deliberative alignment. This
leads to state-of-the-art performance on certain benchmarks for risks such as
generating illicit advice, choosing stereotyped responses, and succumbing to
known jailbreaks. Training models to incorporate a chain of thought before
answering has the potential to unlock substantial benefits, while also
increasing potential risks that stem from heightened intelligence. Our results
underscore the need for building robust alignment methods, extensively
stress-testing their efficacy, and maintaining meticulous risk management
protocols. This report outlines the safety work carried out for the OpenAI o1
and OpenAI o1-mini models, including safety evaluations, external red teaming,
and Preparedness Framework evaluations.


http://arxiv.org/abs/2412.16512
TrojFlow: Flow Models are Natural Targets for Trojan Attacks. (78%)
Zhengyang Qi; Xiaohua Xu
  Flow-based generative models (FMs) have rapidly advanced as a method for
mapping noise to data, its efficient training and sampling process makes it
widely applicable in various fields. FMs can be viewed as a variant of
diffusion models (DMs). At the same time, previous studies have shown that DMs
are vulnerable to Trojan/Backdoor attacks, a type of output manipulation attack
triggered by a maliciously embedded pattern at model input. We found that
Trojan attacks on generative models are essentially equivalent to image
transfer tasks from the backdoor distribution to the target distribution, the
unique ability of FMs to fit any two arbitrary distributions significantly
simplifies the training and sampling setups for attacking FMs, making them
inherently natural targets for backdoor attacks. In this paper, we propose
TrojFlow, exploring the vulnerabilities of FMs through Trojan attacks. In
particular, we consider various attack settings and their combinations and
thoroughly explore whether existing defense methods for DMs can effectively
defend against our proposed attack scenarios. We evaluate TrojFlow on CIFAR-10
and CelebA datasets, our experiments show that our method can compromise FMs
with high utility and specificity, and can easily break through existing
defense mechanisms.


http://arxiv.org/abs/2412.16708
Towards More Robust Retrieval-Augmented Generation: Evaluating RAG Under Adversarial Poisoning Attacks. (76%)
Jinyan Su; Jin Peng Zhou; Zhengxin Zhang; Preslav Nakov; Claire Cardie
  Retrieval-Augmented Generation (RAG) systems have emerged as a promising
solution to mitigate LLM hallucinations and enhance their performance in
knowledge-intensive domains. However, these systems are vulnerable to
adversarial poisoning attacks, where malicious passages injected into retrieval
databases can mislead the model into generating factually incorrect outputs. In
this paper, we investigate both the retrieval and the generation components of
RAG systems to understand how to enhance their robustness against such attacks.
From the retrieval perspective, we analyze why and how the adversarial contexts
are retrieved and assess how the quality of the retrieved passages impacts
downstream generation. From a generation perspective, we evaluate whether LLMs'
advanced critical thinking and internal knowledge capabilities can be leveraged
to mitigate the impact of adversarial contexts, i.e., using skeptical prompting
as a self-defense mechanism. Our experiments and findings provide actionable
insights into designing safer and more resilient retrieval-augmented
frameworks, paving the way for their reliable deployment in real-world
applications.


http://arxiv.org/abs/2412.16633
POEX: Policy Executable Embodied AI Jailbreak Attacks. (9%)
Xuancun Lu; Zhengxian Huang; Xinfeng Li; Xiaoyu ji; Wenyuan Xu
  The integration of large language models (LLMs) into the planning module of
Embodied Artificial Intelligence (Embodied AI) systems has greatly enhanced
their ability to translate complex user instructions into executable policies.
In this paper, we demystified how traditional LLM jailbreak attacks behave in
the Embodied AI context. We conducted a comprehensive safety analysis of the
LLM-based planning module of embodied AI systems against jailbreak attacks.
Using the carefully crafted Harmful-RLbench, we accessed 20 open-source and
proprietary LLMs under traditional jailbreak attacks, and highlighted two key
challenges when adopting the prior jailbreak techniques to embodied AI
contexts: (1) The harmful text output by LLMs does not necessarily induce
harmful policies in Embodied AI context, and (2) even we can generate harmful
policies, we have to guarantee they are executable in practice. To overcome
those challenges, we propose Policy Executable (POEX) jailbreak attacks, where
harmful instructions and optimized suffixes are injected into LLM-based
planning modules, leading embodied AI to perform harmful actions in both
simulated and physical environments. Our approach involves constraining
adversarial suffixes to evade detection and fine-tuning a policy evaluater to
improve the executability of harmful policies. We conducted extensive
experiments on both a robotic arm embodied AI platform and simulators, to
validate the attack and policy success rates on 136 harmful instructions from
Harmful-RLbench. Our findings expose serious safety vulnerabilities in
LLM-based planning modules, including the ability of POEX to be transferred
across models. Finally, we propose mitigation strategies, such as
safety-constrained prompts, pre- and post-planning checks, to address these
vulnerabilities and ensure the safe deployment of embodied AI in real-world
settings.


http://arxiv.org/abs/2412.16780
Forget Vectors at Play: Universal Input Perturbations Driving Machine Unlearning in Image Classification. (8%)
Changchang Sun; Ren Wang; Yihua Zhang; Jinghan Jia; Jiancheng Liu; Gaowen Liu; Sijia Liu; Yan Yan
  Machine unlearning (MU), which seeks to erase the influence of specific
unwanted data from already-trained models, is becoming increasingly vital in
model editing, particularly to comply with evolving data regulations like the
``right to be forgotten''. Conventional approaches are predominantly
model-based, typically requiring retraining or fine-tuning the model's weights
to meet unlearning requirements. In this work, we approach the MU problem from
a novel input perturbation-based perspective, where the model weights remain
intact throughout the unlearning process. We demonstrate the existence of a
proactive input-based unlearning strategy, referred to forget vector, which can
be generated as an input-agnostic data perturbation and remains as effective as
model-based approximate unlearning approaches. We also explore forget vector
arithmetic, whereby multiple class-specific forget vectors are combined through
simple operations (e.g., linear combinations) to generate new forget vectors
for unseen unlearning tasks, such as forgetting arbitrary subsets across
classes. Extensive experiments validate the effectiveness and adaptability of
the forget vector, showcasing its competitive performance relative to
state-of-the-art model-based methods. Codes are available at
https://github.com/Changchangsun/Forget-Vector.


http://arxiv.org/abs/2412.16682
The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents. (4%)
Feiran Jia; Tong Wu; Xin Qin; Anna Squicciarini
  Large Language Model (LLM) agents are increasingly being deployed as
conversational assistants capable of performing complex real-world tasks
through tool integration. This enhanced ability to interact with external
systems and process various data sources, while powerful, introduces
significant security vulnerabilities. In particular, indirect prompt injection
attacks pose a critical threat, where malicious instructions embedded within
external data sources can manipulate agents to deviate from user intentions.
While existing defenses based on rule constraints, source spotlighting, and
authentication protocols show promise, they struggle to maintain robust
security while preserving task functionality. We propose a novel and orthogonal
perspective that reframes agent security from preventing harmful actions to
ensuring task alignment, requiring every agent action to serve user objectives.
Based on this insight, we develop Task Shield, a test-time defense mechanism
that systematically verifies whether each instruction and tool call contributes
to user-specified goals. Through experiments on the AgentDojo benchmark, we
demonstrate that Task Shield reduces attack success rates (2.07\%) while
maintaining high task utility (69.79\%) on GPT-4o.


http://arxiv.org/abs/2412.19834
RoboSignature: Robust Signature and Watermarking on Network Attacks. (1%)
Aryaman Shaan; Garvit Banga; Raghav Mantri
  Generative models have enabled easy creation and generation of images of all
kinds given a single prompt. However, this has also raised ethical concerns
about what is an actual piece of content created by humans or cameras compared
to model-generated content like images or videos. Watermarking data generated
by modern generative models is a popular method to provide information on the
source of the content. The goal is for all generated images to conceal an
invisible watermark, allowing for future detection or identification. The
Stable Signature finetunes the decoder of Latent Diffusion Models such that a
unique watermark is rooted in any image produced by the decoder. In this paper,
we present a novel adversarial fine-tuning attack that disrupts the model's
ability to embed the intended watermark, exposing a significant vulnerability
in existing watermarking methods. To address this, we further propose a
tamper-resistant fine-tuning algorithm inspired by methods developed for large
language models, tailored to the specific requirements of watermarking in LDMs.
Our findings emphasize the importance of anticipating and defending against
potential vulnerabilities in generative systems.


http://arxiv.org/abs/2412.16254
Adversarial Robustness through Dynamic Ensemble Learning. (99%)
Hetvi Waghela; Jaydip Sen; Sneha Rakshit
  Adversarial attacks pose a significant threat to the reliability of
pre-trained language models (PLMs) such as GPT, BERT, RoBERTa, and T5. This
paper presents Adversarial Robustness through Dynamic Ensemble Learning
(ARDEL), a novel scheme designed to enhance the robustness of PLMs against such
attacks. ARDEL leverages the diversity of multiple PLMs and dynamically adjusts
the ensemble configuration based on input characteristics and detected
adversarial patterns. Key components of ARDEL include a meta-model for dynamic
weighting, an adversarial pattern detection module, and adversarial training
with regularization techniques. Comprehensive evaluations using standardized
datasets and various adversarial attack scenarios demonstrate that ARDEL
significantly improves robustness compared to existing methods. By dynamically
reconfiguring the ensemble to prioritize the most robust models for each input,
ARDEL effectively reduces attack success rates and maintains higher accuracy
under adversarial conditions. This work contributes to the broader goal of
developing more secure and trustworthy AI systems for real-world NLP
applications, offering a practical and scalable solution to enhance adversarial
resilience in PLMs.


http://arxiv.org/abs/2412.16382
EMPRA: Embedding Perturbation Rank Attack against Neural Ranking Models. (98%)
Amin Bigdeli; Negar Arabzadeh; Ebrahim Bagheri; Charles L. A. Clarke
  Recent research has shown that neural information retrieval techniques may be
susceptible to adversarial attacks. Adversarial attacks seek to manipulate the
ranking of documents, with the intention of exposing users to targeted content.
In this paper, we introduce the Embedding Perturbation Rank Attack (EMPRA)
method, a novel approach designed to perform adversarial attacks on black-box
Neural Ranking Models (NRMs). EMPRA manipulates sentence-level embeddings,
guiding them towards pertinent context related to the query while preserving
semantic integrity. This process generates adversarial texts that seamlessly
integrate with the original content and remain imperceptible to humans. Our
extensive evaluation conducted on the widely-used MS MARCO V1 passage
collection demonstrate the effectiveness of EMPRA against a wide range of
state-of-the-art baselines in promoting a specific set of target documents
within a given ranked results. Specifically, EMPRA successfully achieves a
re-ranking of almost 96% of target documents originally ranked between 51-100
to rank within the top 10. Furthermore, EMPRA does not depend on surrogate
models for adversarial text generation, enhancing its robustness against
different NRMs in realistic settings.


http://arxiv.org/abs/2412.15924
Watertox: The Art of Simplicity in Universal Attacks A Cross-Model Framework for Robust Adversarial Generation. (98%)
Zhenghao Gao; Shengjie Xu; Meixi Chen; Fangyao Zhao
  Contemporary adversarial attack methods face significant limitations in
cross-model transferability and practical applicability. We present Watertox,
an elegant adversarial attack framework achieving remarkable effectiveness
through architectural diversity and precision-controlled perturbations. Our
two-stage Fast Gradient Sign Method combines uniform baseline perturbations
($\epsilon_1 = 0.1$) with targeted enhancements ($\epsilon_2 = 0.4$). The
framework leverages an ensemble of complementary architectures, from VGG to
ConvNeXt, synthesizing diverse perspectives through an innovative voting
mechanism. Against state-of-the-art architectures, Watertox reduces model
accuracy from 70.6% to 16.0%, with zero-shot attacks achieving up to 98.8%
accuracy reduction against unseen architectures. These results establish
Watertox as a significant advancement in adversarial methodologies, with
promising applications in visual security systems and CAPTCHA generation.


http://arxiv.org/abs/2412.15623
JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMs. (41%)
Hongyi Li; Jiawei Ye; Jie Wu; Tianjie Yan; Chu Wang; Zhixin Li
  Large Language Models (LLMs) aligned with human feedback have recently
garnered significant attention. However, it remains vulnerable to jailbreak
attacks, where adversaries manipulate prompts to induce harmful outputs.
Exploring jailbreak attacks enables us to investigate the vulnerabilities of
LLMs and further guides us in enhancing their security. Unfortunately, existing
techniques mainly rely on handcrafted templates or generated-based
optimization, posing challenges in scalability, efficiency and universality. To
address these issues, we present JailPO, a novel black-box jailbreak framework
to examine LLM alignment. For scalability and universality, JailPO meticulously
trains attack models to automatically generate covert jailbreak prompts.
Furthermore, we introduce a preference optimization-based attack method to
enhance the jailbreak effectiveness, thereby improving efficiency. To analyze
model vulnerabilities, we provide three flexible jailbreak patterns. Extensive
experiments demonstrate that JailPO not only automates the attack process while
maintaining effectiveness but also exhibits superior performance in efficiency,
universality, and robustness against defenses compared to baselines.
Additionally, our analysis of the three JailPO patterns reveals that attacks
based on complex templates exhibit higher attack strength, whereas covert
question transformations elicit riskier responses and are more likely to bypass
defense mechanisms.


http://arxiv.org/abs/2412.15614
Technical Report for ICML 2024 TiFA Workshop MLLM Attack Challenge: Suffix Injection and Projected Gradient Descent Can Easily Fool An MLLM. (13%)
Yangyang Guo; Ziwei Xu; Xilie Xu; YongKang Wong; Liqiang Nie; Mohan Kankanhalli
  This technical report introduces our top-ranked solution that employs two
approaches, \ie suffix injection and projected gradient descent (PGD) , to
address the TiFA workshop MLLM attack challenge. Specifically, we first append
the text from an incorrectly labeled option (pseudo-labeled) to the original
query as a suffix. Using this modified query, our second approach applies the
PGD method to add imperceptible perturbations to the image. Combining these two
techniques enables successful attacks on the LLaVA 1.5 model.


http://arxiv.org/abs/2412.16358
Texture- and Shape-based Adversarial Attacks for Vehicle Detection in Synthetic Overhead Imagery. (9%)
Mikael Yeghiazaryan; Sai Abhishek Siddhartha Namburu; Emily Kim; Stanislav Panev; Melo Celso de; Brent Lance; la Torre Fernando De; Jessica K. Hodgins
  Detecting vehicles in aerial images can be very challenging due to complex
backgrounds, small resolution, shadows, and occlusions. Despite the
effectiveness of SOTA detectors such as YOLO, they remain vulnerable to
adversarial attacks (AAs), compromising their reliability. Traditional AA
strategies often overlook the practical constraints of physical implementation,
focusing solely on attack performance. Our work addresses this issue by
proposing practical implementation constraints for AA in texture and/or shape.
These constraints include pixelation, masking, limiting the color palette of
the textures, and constraining the shape modifications. We evaluated the
proposed constraints through extensive experiments using three widely used
object detector architectures, and compared them to previous works. The results
demonstrate the effectiveness of our solutions and reveal a trade-off between
practicality and performance. Additionally, we introduce a labeled dataset of
overhead images featuring vehicles of various categories. We will make the
code/dataset public upon paper acceptance.


http://arxiv.org/abs/2412.15704
PoisonCatcher: Revealing and Identifying LDP Poisoning Attacks in IIoT. (9%)
Lisha Shuai; Shaofeng Tan; Nan Zhang; Jiamin Zhang; Min Zhang; Xiaolong Yang
  Local Differential Privacy (LDP) is widely adopted in the Industrial Internet
of Things (IIoT) for its lightweight, decentralized, and scalable nature.
However, its perturbation-based privacy mechanism makes it difficult to
distinguish between uncontaminated and tainted data, encouraging adversaries to
launch poisoning attacks. While LDP provides some resilience against minor
poisoning, it lacks robustness in IIoT with dynamic networks and substantial
real-time data flows. Effective countermeasures for such attacks are still
underdeveloped. This work narrows the critical gap by revealing and identifying
LDP poisoning attacks in IIoT. We begin by deepening the understanding of such
attacks, revealing novel threats that arise from the interplay between LDP
indistinguishability and IIoT complexity. This exploration uncovers a novel
rule-poisoning attack, and presents a general attack formulation by unifying it
with input-poisoning and output-poisoning. Furthermore, two key attack impacts,
i.e., Statistical Query Result (SQR) accuracy degradation and inter-dataset
correlations disruption, along with two characteristics: attack patterns
unstable and poisoned data stealth are revealed. From this, we propose
PoisonCatcher, a four-stage solution that detects LDP poisoning attacks and
identifies specific contaminated data points. It utilizes temporal similarity,
attribute correlation, and time-series stability analysis to detect datasets
exhibiting SQR accuracy degradation, inter-dataset disruptions, and unstable
patterns. Enhanced feature engineering is used to extract subtle poisoning
signatures, enabling machine learning models to identify specific
contamination. Experimental evaluations show the effectiveness, achieving
state-of-the-art performance with average precision and recall rates of 86.17%
and 97.5%, respectively, across six representative attack scenarios.


http://arxiv.org/abs/2412.16457
Robust random graph matching in dense graphs via vector approximate message passing. (1%)
Zhangsong Li
  In this paper, we focus on the matching recovery problem between a pair of
correlated Gaussian Wigner matrices with a latent vertex correspondence. We are
particularly interested in a robust version of this problem such that our
observation is a perturbed input $(A+E,B+F)$ where $(A,B)$ is a pair of
correlated Gaussian Wigner matrices and $E,F$ are adversarially chosen matrices
supported on an unknown $\epsilon n * \epsilon n$ principle minor of $A,B$,
respectively. We propose a vector-approximate message passing (vector-AMP)
algorithm that succeeds in polynomial time as long as the correlation $\rho$
between $(A,B)$ is a non-vanishing constant and $\epsilon = o\big(
\tfrac{1}{(\log n)^{20}} \big)$.
  The main methodological inputs for our result are the iterative random graph
matching algorithm proposed in \cite{DL22+, DL23+} and the spectral cleaning
procedure proposed in \cite{IS24+}. To the best of our knowledge, our algorithm
is the first efficient random graph matching type algorithm that is robust
under any adversarial perturbations of $n^{1-o(1)}$ size.


http://arxiv.org/abs/2412.15206
AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Driving. (5%)
Shuo Xing; Hongyuan Hua; Xiangbo Gao; Shenzhe Zhu; Renjie Li; Kexin Tian; Xiaopeng Li; Heng Huang; Tianbao Yang; Zhangyang Wang; Yang Zhou; Huaxiu Yao; Zhengzhong Tu
  Recent advancements in large vision language models (VLMs) tailored for
autonomous driving (AD) have shown strong scene understanding and reasoning
capabilities, making them undeniable candidates for end-to-end driving systems.
However, limited work exists on studying the trustworthiness of DriveVLMs -- a
critical factor that directly impacts public transportation safety. In this
paper, we introduce AutoTrust, a comprehensive trustworthiness benchmark for
large vision-language models in autonomous driving (DriveVLMs), considering
diverse perspectives -- including trustfulness, safety, robustness, privacy,
and fairness. We constructed the largest visual question-answering dataset for
investigating trustworthiness issues in driving scenarios, comprising over 10k
unique scenes and 18k queries. We evaluated six publicly available VLMs,
spanning from generalist to specialist, from open-source to commercial models.
Our exhaustive evaluations have unveiled previously undiscovered
vulnerabilities of DriveVLMs to trustworthiness threats. Specifically, we found
that the general VLMs like LLaVA-v1.6 and GPT-4o-mini surprisingly outperform
specialized models fine-tuned for driving in terms of overall trustworthiness.
DriveVLMs like DriveLM-Agent are particularly vulnerable to disclosing
sensitive information. Additionally, both generalist and specialist VLMs remain
susceptible to adversarial attacks and struggle to ensure unbiased
decision-making across diverse environments and populations. Our findings call
for immediate and decisive action to address the trustworthiness of DriveVLMs
-- an issue of critical importance to public safety and the welfare of all
citizens relying on autonomous transportation systems. Our benchmark is
publicly available at \url{https://github.com/taco-group/AutoTrust}, and the
leaderboard is released at \url{https://taco-group.github.io/AutoTrust/}.


http://arxiv.org/abs/2412.14738
Boosting GNN Performance via Training Sample Selection Based on Adversarial Robustness Evaluation. (4%)
Yongyu Wang
  Graph Neural Networks (GNNs) have established themselves as one of the most
powerful neural network architectures, excelling in leveraging graph topology
and node features for various tasks. However, GNNs are inherently vulnerable to
noise in their inputs. Such noise can significantly degrade their performance.
To address this challenge, we propose a novel approach that employs adversarial
robustness evaluation techniques to identify nodes in the graph that are most
susceptible to noise. By selecting and constructing a training set composed of
these particularly noise-prone nodes, we then use them to train a Graph
Convolutional Network (GCN). Our experimental results demonstrate that this
strategy leads to substantial improvements in the GCN's performance.


http://arxiv.org/abs/2412.13880
A Review of the Duality of Adversarial Learning in Network Intrusion: Attacks and Countermeasures. (89%)
Shalini Saini; Anitha Chennamaneni; Babatunde Sawyerr
  Deep learning solutions are instrumental in cybersecurity, harnessing their
ability to analyze vast datasets, identify complex patterns, and detect
anomalies. However, malevolent actors can exploit these capabilities to
orchestrate sophisticated attacks, posing significant challenges to defenders
and traditional security measures. Adversarial attacks, particularly those
targeting vulnerabilities in deep learning models, present a nuanced and
substantial threat to cybersecurity. Our study delves into adversarial learning
threats such as Data Poisoning, Test Time Evasion, and Reverse Engineering,
specifically impacting Network Intrusion Detection Systems. Our research
explores the intricacies and countermeasures of attacks to deepen understanding
of network security challenges amidst adversarial threats. In our study, we
present insights into the dynamic realm of adversarial learning and its
implications for network intrusion. The intersection of adversarial attacks and
defenses within network traffic data, coupled with advances in machine learning
and deep learning techniques, represents a relatively underexplored domain. Our
research lays the groundwork for strengthening defense mechanisms to address
the potential breaches in network security and privacy posed by adversarial
attacks. Through our in-depth analysis, we identify domain-specific research
gaps, such as the scarcity of real-life attack data and the evaluation of
AI-based solutions for network traffic. Our focus on these challenges aims to
stimulate future research efforts toward the development of resilient network
defense strategies.


http://arxiv.org/abs/2412.13705
Mitigating Adversarial Attacks in LLMs through Defensive Suffix Generation. (83%)
Minkyoung Kim; Yunha Kim; Hyeram Seo; Heejung Choi; Jiye Han; Gaeun Kee; Soyoung Ko; HyoJe Jung; Byeolhee Kim; Young-Hak Kim; Sanghyun Park; Tae Joon Jun
  Large language models (LLMs) have exhibited outstanding performance in
natural language processing tasks. However, these models remain susceptible to
adversarial attacks in which slight input perturbations can lead to harmful or
misleading outputs. A gradient-based defensive suffix generation algorithm is
designed to bolster the robustness of LLMs. By appending carefully optimized
defensive suffixes to input prompts, the algorithm mitigates adversarial
influences while preserving the models' utility. To enhance adversarial
understanding, a novel total loss function ($L_{\text{total}}$) combining
defensive loss ($L_{\text{def}}$) and adversarial loss ($L_{\text{adv}}$)
generates defensive suffixes more effectively. Experimental evaluations
conducted on open-source LLMs such as Gemma-7B, mistral-7B, Llama2-7B, and
Llama2-13B show that the proposed method reduces attack success rates (ASR) by
an average of 11\% compared to models without defensive suffixes. Additionally,
the perplexity score of Gemma-7B decreased from 6.57 to 3.93 when applying the
defensive suffix generated by openELM-270M. Furthermore, TruthfulQA evaluations
demonstrate consistent improvements with Truthfulness scores increasing by up
to 10\% across tested configurations. This approach significantly enhances the
security of LLMs in critical applications without requiring extensive
retraining.


http://arxiv.org/abs/2412.13709
Physics-Based Adversarial Attack on Near-Infrared Human Detector for Nighttime Surveillance Camera Systems. (78%)
Muyao Niu; Zhuoxiao Li; Yifan Zhan; Huy H. Nguyen; Isao Echizen; Yinqiang Zheng
  Many surveillance cameras switch between daytime and nighttime modes based on
illuminance levels. During the day, the camera records ordinary RGB images
through an enabled IR-cut filter. At night, the filter is disabled to capture
near-infrared (NIR) light emitted from NIR LEDs typically mounted around the
lens. While RGB-based AI algorithm vulnerabilities have been widely reported,
the vulnerabilities of NIR-based AI have rarely been investigated. In this
paper, we identify fundamental vulnerabilities in NIR-based image understanding
caused by color and texture loss due to the intrinsic characteristics of
clothes' reflectance and cameras' spectral sensitivity in the NIR range. We
further show that the nearly co-located configuration of illuminants and
cameras in existing surveillance systems facilitates concealing and fully
passive attacks in the physical world. Specifically, we demonstrate how
retro-reflective and insulation plastic tapes can manipulate the intensity
distribution of NIR images. We showcase an attack on the YOLO-based human
detector using binary patterns designed in the digital space (via black-box
query and searching) and then physically realized using tapes pasted onto
clothes. Our attack highlights significant reliability concerns for nighttime
surveillance systems, which are intended to enhance security. Codes Available:
https://github.com/MyNiuuu/AdvNIR


http://arxiv.org/abs/2412.13879
Crabs: Consuming Resrouce via Auto-generation for LLM-DoS Attack under Black-box Settings. (75%)
Yuanhe Zhang; Zhenhong Zhou; Wei Zhang; Xinyue Wang; Xiaojun Jia; Yang Liu; Sen Su
  Large Language Models (LLMs) have demonstrated remarkable performance across
diverse tasks. LLMs continue to be vulnerable to external threats, particularly
Denial-of-Service (DoS) attacks. Specifically, LLM-DoS attacks aim to exhaust
computational resources and block services. However, prior works tend to focus
on performing white-box attacks, overlooking black-box settings. In this work,
we propose an automated algorithm designed for black-box LLMs, called
Auto-Generation for LLM-DoS Attack (AutoDoS). AutoDoS introduces DoS Attack
Tree and optimizes the prompt node coverage to enhance effectiveness under
black-box conditions. Our method can bypass existing defense with enhanced
stealthiness via semantic improvement of prompt nodes. Furthermore, we reveal
that implanting Length Trojan in Basic DoS Prompt aids in achieving higher
attack efficacy. Experimental results show that AutoDoS amplifies service
response latency by over 250 $\times \uparrow$, leading to severe resource
consumption in terms of GPU utilization and memory usage. Our code is available
at \url{https://github.com/shuita2333/AutoDoS}.


http://arxiv.org/abs/2412.14113
Adversarial Hubness in Multi-Modal Retrieval. (74%)
Tingwei Zhang; Fnu Suya; Rishi Jha; Collin Zhang; Vitaly Shmatikov
  Hubness is a phenomenon in high-dimensional vector spaces where a single
point from the natural distribution is unusually close to many other points.
This is a well-known problem in information retrieval that causes some items to
accidentally (and incorrectly) appear relevant to many queries. In this paper,
we investigate how attackers can exploit hubness to turn any image or audio
input in a multi-modal retrieval system into an adversarial hub. Adversarial
hubs can be used to inject universal adversarial content (e.g., spam) that will
be retrieved in response to thousands of different queries, as well as for
targeted attacks on queries related to specific, attacker-chosen concepts. We
present a method for creating adversarial hubs and evaluate the resulting hubs
on benchmark multi-modal retrieval datasets and an image-to-image retrieval
system based on a tutorial from Pinecone, a popular vector database. For
example, in text-caption-to-image retrieval, a single adversarial hub is
retrieved as the top-1 most relevant image for more than 21,000 out of 25,000
test queries (by contrast, the most common natural hub is the top-1 response to
only 102 queries). We also investigate whether techniques for mitigating
natural hubness are an effective defense against adversarial hubs, and show
that they are not effective against hubs that target queries related to
specific concepts.


http://arxiv.org/abs/2412.13762
Cultivating Archipelago of Forests: Evolving Robust Decision Trees through Island Coevolution. (56%)
Adam Żychowski; Andrew Perrault; Jacek Mańdziuk
  Decision trees are widely used in machine learning due to their simplicity
and interpretability, but they often lack robustness to adversarial attacks and
data perturbations. The paper proposes a novel island-based coevolutionary
algorithm (ICoEvoRDF) for constructing robust decision tree ensembles. The
algorithm operates on multiple islands, each containing populations of decision
trees and adversarial perturbations. The populations on each island evolve
independently, with periodic migration of top-performing decision trees between
islands. This approach fosters diversity and enhances the exploration of the
solution space, leading to more robust and accurate decision tree ensembles.
ICoEvoRDF utilizes a popular game theory concept of mixed Nash equilibrium for
ensemble weighting, which further leads to improvement in results. ICoEvoRDF is
evaluated on 20 benchmark datasets, demonstrating its superior performance
compared to state-of-the-art methods in optimizing both adversarial accuracy
and minimax regret. The flexibility of ICoEvoRDF allows for the integration of
decision trees from various existing methods, providing a unified framework for
combining diverse solutions. Our approach offers a promising direction for
developing robust and interpretable machine learning models


http://arxiv.org/abs/2412.13913
A Black-Box Evaluation Framework for Semantic Robustness in Bird's Eye View Detection. (8%)
Fu Wang; Yanghao Zhang; Xiangyu Yin; Guangliang Cheng; Zeyu Fu; Xiaowei Huang; Wenjie Ruan
  Camera-based Bird's Eye View (BEV) perception models receive increasing
attention for their crucial role in autonomous driving, a domain where concerns
about the robustness and reliability of deep learning have been raised. While
only a few works have investigated the effects of randomly generated semantic
perturbations, aka natural corruptions, on the multi-view BEV detection task,
we develop a black-box robustness evaluation framework that adversarially
optimises three common semantic perturbations: geometric transformation, colour
shifting, and motion blur, to deceive BEV models, serving as the first approach
in this emerging field. To address the challenge posed by optimising the
semantic perturbation, we design a smoothed, distance-based surrogate function
to replace the mAP metric and introduce SimpleDIRECT, a deterministic
optimisation algorithm that utilises observed slopes to guide the optimisation
process. By comparing with randomised perturbation and two optimisation
baselines, we demonstrate the effectiveness of the proposed framework.
Additionally, we provide a benchmark on the semantic robustness of ten recent
BEV models. The results reveal that PolarFormer, which emphasises geometric
information from multi-view images, exhibits the highest robustness, whereas
BEVDet is fully compromised, with its precision reduced to zero.


http://arxiv.org/abs/2412.13917
Speech Watermarking with Discrete Intermediate Representations. (4%)
Shengpeng Ji; Ziyue Jiang; Jialong Zuo; Minghui Fang; Yifu Chen; Tao Jin; Zhou Zhao
  Speech watermarking techniques can proactively mitigate the potential harmful
consequences of instant voice cloning techniques. These techniques involve the
insertion of signals into speech that are imperceptible to humans but can be
detected by algorithms. Previous approaches typically embed watermark messages
into continuous space. However, intuitively, embedding watermark information
into robust discrete latent space can significantly improve the robustness of
watermarking systems. In this paper, we propose DiscreteWM, a novel speech
watermarking framework that injects watermarks into the discrete intermediate
representations of speech. Specifically, we map speech into discrete latent
space with a vector-quantized autoencoder and inject watermarks by changing the
modular arithmetic relation of discrete IDs. To ensure the imperceptibility of
watermarks, we also propose a manipulator model to select the candidate tokens
for watermark embedding. Experimental results demonstrate that our framework
achieves state-of-the-art performance in robustness and imperceptibility,
simultaneously. Moreover, our flexible frame-wise approach can serve as an
efficient solution for both voice cloning detection and information hiding.
Additionally, DiscreteWM can encode 1 to 150 bits of watermark information
within a 1-second speech clip, indicating its encoding capacity. Audio samples
are available at https://DiscreteWM.github.io/discrete_wm.


http://arxiv.org/abs/2412.14080
On the Robustness of Distributed Machine Learning against Transfer Attacks. (2%)
Sébastien Andreina; Pascal Zimmer; Ghassan Karame
  Although distributed machine learning (distributed ML) is gaining
considerable attention in the community, prior works have independently looked
at instances of distributed ML in either the training or the inference phase.
No prior work has examined the combined robustness stemming from distributing
both the learning and the inference process. In this work, we explore, for the
first time, the robustness of distributed ML models that are fully
heterogeneous in training data, architecture, scheduler, optimizer, and other
model parameters. Supported by theory and extensive experimental validation
using CIFAR10 and FashionMNIST, we show that such properly distributed ML
instantiations achieve across-the-board improvements in accuracy-robustness
tradeoffs against state-of-the-art transfer-based attacks that could otherwise
not be realized by current ensemble or federated learning instantiations. For
instance, our experiments on CIFAR10 show that for the Common Weakness attack,
one of the most powerful state-of-the-art transfer-based attacks, our method
improves robust accuracy by up to 40%, with a minimal impact on clean task
accuracy.


http://arxiv.org/abs/2412.13753
Mesoscopic Insights: Orchestrating Multi-scale & Hybrid Architecture for Image Manipulation Localization. (1%)
Xuekang Zhu; Xiaochen Ma; Lei Su; Zhuohang Jiang; Bo Du; Xiwen Wang; Zeyu Lei; Wentao Feng; Chi-Man Pun; Jizhe Zhou
  The mesoscopic level serves as a bridge between the macroscopic and
microscopic worlds, addressing gaps overlooked by both. Image manipulation
localization (IML), a crucial technique to pursue truth from fake images, has
long relied on low-level (microscopic-level) traces. However, in practice, most
tampering aims to deceive the audience by altering image semantics. As a
result, manipulation commonly occurs at the object level (macroscopic level),
which is equally important as microscopic traces. Therefore, integrating these
two levels into the mesoscopic level presents a new perspective for IML
research. Inspired by this, our paper explores how to simultaneously construct
mesoscopic representations of micro and macro information for IML and
introduces the Mesorch architecture to orchestrate both. Specifically, this
architecture i) combines Transformers and CNNs in parallel, with Transformers
extracting macro information and CNNs capturing micro details, and ii) explores
across different scales, assessing micro and macro information seamlessly.
Additionally, based on the Mesorch architecture, the paper introduces two
baseline models aimed at solving IML tasks through mesoscopic representation.
Extensive experiments across four datasets have demonstrated that our models
surpass the current state-of-the-art in terms of performance, computational
complexity, and robustness.


http://arxiv.org/abs/2412.13525
Hybrid Data-Free Knowledge Distillation. (1%)
Jialiang Tang; Shuo Chen; Chen Gong
  Data-free knowledge distillation aims to learn a compact student network from
a pre-trained large teacher network without using the original training data of
the teacher network. Existing collection-based and generation-based methods
train student networks by collecting massive real examples and generating
synthetic examples, respectively. However, they inevitably become weak in
practical scenarios due to the difficulties in gathering or emulating
sufficient real-world data. To solve this problem, we propose a novel method
called \textbf{H}ybr\textbf{i}d \textbf{D}ata-\textbf{F}ree
\textbf{D}istillation (HiDFD), which leverages only a small amount of collected
data as well as generates sufficient examples for training student networks.
Our HiDFD comprises two primary modules, \textit{i.e.}, the teacher-guided
generation and student distillation. The teacher-guided generation module
guides a Generative Adversarial Network (GAN) by the teacher network to produce
high-quality synthetic examples from very few real-world collected examples.
Specifically, we design a feature integration mechanism to prevent the GAN from
overfitting and facilitate the reliable representation learning from the
teacher network. Meanwhile, we drive a category frequency smoothing technique
via the teacher network to balance the generative training of each category. In
the student distillation module, we explore a data inflation strategy to
properly utilize a blend of real and synthetic data to train the student
network via a classifier-sharing-based feature alignment technique. Intensive
experiments across multiple benchmarks demonstrate that our HiDFD can achieve
state-of-the-art performance using 120 times less collected data than existing
methods. Code is available at https://github.com/tangjialiang97/HiDFD.


http://arxiv.org/abs/2412.13866
SHAP scores fail pervasively even when Lipschitz succeeds. (1%)
Olivier Letoffe; Xuanxiang Huang; Joao Marques-Silva
  The ubiquitous use of Shapley values in eXplainable AI (XAI) has been
triggered by the tool SHAP, and as a result are commonly referred to as SHAP
scores. Recent work devised examples of machine learning (ML) classifiers for
which the computed SHAP scores are thoroughly unsatisfactory, by allowing human
decision-makers to be misled. Nevertheless, such examples could be perceived as
somewhat artificial, since the selected classes must be interpreted as numeric.
Furthermore, it was unclear how general were the issues identified with SHAP
scores. This paper answers these criticisms. First, the paper shows that for
Boolean classifiers there are arbitrarily many examples for which the SHAP
scores must be deemed unsatisfactory. Second, the paper shows that the issues
with SHAP scores are also observed in the case of regression models. In
addition, the paper studies the class of regression models that respect
Lipschitz continuity, a measure of a function's rate of change that finds
important recent uses in ML, including model robustness. Concretely, the paper
shows that the issues with SHAP scores occur even for regression models that
respect Lipschitz continuity. Finally, the paper shows that the same issues are
guaranteed to exist for arbitrarily differentiable regression models.


http://arxiv.org/abs/2412.13507
Novel AI Camera Camouflage: Face Cloaking Without Full Disguise. (1%)
David Noever; Forrest McKee
  This study demonstrates a novel approach to facial camouflage that combines
targeted cosmetic perturbations and alpha transparency layer manipulation to
evade modern facial recognition systems. Unlike previous methods -- such as CV
dazzle, adversarial patches, and theatrical disguises -- this work achieves
effective obfuscation through subtle modifications to key-point regions,
particularly the brow, nose bridge, and jawline. Empirical testing with Haar
cascade classifiers and commercial systems like BetaFaceAPI and Microsoft Bing
Visual Search reveals that vertical perturbations near dense facial key points
significantly disrupt detection without relying on overt disguises.
Additionally, leveraging alpha transparency attacks in PNG images creates a
dual-layer effect: faces remain visible to human observers but disappear in
machine-readable RGB layers, rendering them unidentifiable during reverse image
searches. The results highlight the potential for creating scalable,
low-visibility facial obfuscation strategies that balance effectiveness and
subtlety, opening pathways for defeating surveillance while maintaining
plausible anonymity.


http://arxiv.org/abs/2412.12626
Improving the Transferability of 3D Point Cloud Attack via Spectral-aware Admix and Optimization Designs. (99%)
Shiyu Hu; Daizong Liu; Wei Hu
  Deep learning models for point clouds have shown to be vulnerable to
adversarial attacks, which have received increasing attention in various
safety-critical applications such as autonomous driving, robotics, and
surveillance. Existing 3D attackers generally design various attack strategies
in the white-box setting, requiring the prior knowledge of 3D model details.
However, real-world 3D applications are in the black-box setting, where we can
only acquire the outputs of the target classifier. Although few recent works
try to explore the black-box attack, they still achieve limited attack success
rates (ASR). To alleviate this issue, this paper focuses on attacking the 3D
models in a transfer-based black-box setting, where we first carefully design
adversarial examples in a white-box surrogate model and then transfer them to
attack other black-box victim models. Specifically, we propose a novel
Spectral-aware Admix with Augmented Optimization method (SAAO) to improve the
adversarial transferability. In particular, since traditional Admix strategy
are deployed in the 2D domain that adds pixel-wise images for perturbing, we
can not directly follow it to merge point clouds in coordinate domain as it
will destroy the geometric shapes. Therefore, we design spectral-aware fusion
that performs Graph Fourier Transform (GFT) to get spectral features of the
point clouds and add them in the spectral domain. Afterward, we run a few steps
with spectral-aware weighted Admix to select better optimization paths as well
as to adjust corresponding learning weights. At last, we run more steps to
generate adversarial spectral feature along the optimization path and perform
Inverse-GFT on the adversarial spectral feature to obtain the adversarial
example in the data domain. Experiments show that our SAAO achieves better
transferability compared to existing 3D attack methods.


http://arxiv.org/abs/2412.13376
Targeted View-Invariant Adversarial Perturbations for 3D Object Recognition. (99%)
Christian Green; Mehmet Ergezer; Abdurrahman Zeybey
  Adversarial attacks pose significant challenges in 3D object recognition,
especially in scenarios involving multi-view analysis where objects can be
observed from varying angles. This paper introduces View-Invariant Adversarial
Perturbations (VIAP), a novel method for crafting robust adversarial examples
that remain effective across multiple viewpoints. Unlike traditional methods,
VIAP enables targeted attacks capable of manipulating recognition systems to
classify objects as specific, pre-determined labels, all while using a single
universal perturbation. Leveraging a dataset of 1,210 images across 121 diverse
rendered 3D objects, we demonstrate the effectiveness of VIAP in both targeted
and untargeted settings. Our untargeted perturbations successfully generate a
singular adversarial noise robust to 3D transformations, while targeted attacks
achieve exceptional results, with top-1 accuracies exceeding 95% across various
epsilon values. These findings highlight VIAPs potential for real-world
applications, such as testing the robustness of 3D recognition systems. The
proposed method sets a new benchmark for view-invariant adversarial robustness,
advancing the field of adversarial machine learning for 3D object recognition.


http://arxiv.org/abs/2412.16213
AdvIRL: Reinforcement Learning-Based Adversarial Attacks on 3D NeRF Models. (98%)
Tommy Nguyen; Mehmet Ergezer; Christian Green
  The increasing deployment of AI models in critical applications has exposed
them to significant risks from adversarial attacks. While adversarial
vulnerabilities in 2D vision models have been extensively studied, the threat
landscape for 3D generative models, such as Neural Radiance Fields (NeRF),
remains underexplored. This work introduces \textit{AdvIRL}, a novel framework
for crafting adversarial NeRF models using Instant Neural Graphics Primitives
(Instant-NGP) and Reinforcement Learning. Unlike prior methods, \textit{AdvIRL}
generates adversarial noise that remains robust under diverse 3D
transformations, including rotations and scaling, enabling effective black-box
attacks in real-world scenarios. Our approach is validated across a wide range
of scenes, from small objects (e.g., bananas) to large environments (e.g.,
lighthouses). Notably, targeted attacks achieved high-confidence
misclassifications, such as labeling a banana as a slug and a truck as a
cannon, demonstrating the practical risks posed by adversarial NeRFs. Beyond
attacking, \textit{AdvIRL}-generated adversarial models can serve as
adversarial training data to enhance the robustness of vision systems. The
implementation of \textit{AdvIRL} is publicly available at
\url{https://github.com/Tommy-Nguyen-cpu/AdvIRL/tree/MultiView-Clean}, ensuring
reproducibility and facilitating future research.


http://arxiv.org/abs/2412.12722
Defending LVLMs Against Vision Attacks through Partial-Perception Supervision. (92%)
Qi Zhou; Tianlin Li; Qing Guo; Dongxia Wang; Yun Lin; Yang Liu; Jin Song Dong
  Recent studies have raised significant concerns regarding the vulnerability
of Large Vision Language Models (LVLMs) to maliciously injected or perturbed
input images, which can mislead their responses. Existing defense methods show
that such vision attacks are sensitive to image modifications especially
cropping, using majority voting across responses of modified images as
corrected responses. However, these modifications often result in partial
images and distort the semantics, which reduces response quality on clean
images after voting. Instead of directly using responses from partial images
for voting, we investigate using them to supervise the LVLM's responses to the
original images. We propose a black-box, training-free method called DPS
(Defense through Partial-Perception Supervision). In this approach, the model
is prompted using the responses generated by a model that perceives only a
partial image. With DPS, the model can adjust its response based on partial
image understanding when under attack, while confidently maintaining its
original response for clean input. Our findings show that the weak model can
supervise the strong model: when faced with an attacked input, the strong model
becomes less confident and adjusts its response based on the weak model's
partial understanding, effectively defending against the attack. With clean
input, it confidently maintains its original response. Empirical experiments
show our method outperforms the baseline, cutting the average attack success
rate by 76.3% across six datasets on three popular models.


http://arxiv.org/abs/2412.15276
Exploring Query Efficient Data Generation towards Data-free Model Stealing in Hard Label Setting. (92%)
Gaozheng Pei; Shaojie lyu; Ke Ma; Pinci Yang; Qianqian Xu; Yingfei Sun
  Data-free model stealing involves replicating the functionality of a target
model into a substitute model without accessing the target model's structure,
parameters, or training data. The adversary can only access the target model's
predictions for generated samples. Once the substitute model closely
approximates the behavior of the target model, attackers can exploit its
white-box characteristics for subsequent malicious activities, such as
adversarial attacks. Existing methods within cooperative game frameworks often
produce samples with high confidence for the prediction of the substitute
model, which makes it difficult for the substitute model to replicate the
behavior of the target model. This paper presents a new data-free model
stealing approach called Query Efficient Data Generation (\textbf{QEDG}). We
introduce two distinct loss functions to ensure the generation of sufficient
samples that closely and uniformly align with the target model's decision
boundary across multiple classes. Building on the limitation of current
methods, which typically yield only one piece of supervised information per
query, we propose the query-free sample augmentation that enables the
acquisition of additional supervised information without increasing the number
of queries. Motivated by theoretical analysis, we adopt the consistency rate
metric, which more accurately evaluates the similarity between the substitute
and target models. We conducted extensive experiments to verify the
effectiveness of our proposed method, which achieved better performance with
fewer queries compared to the state-of-the-art methods on the real
\textbf{MLaaS} scenario and five datasets.


http://arxiv.org/abs/2412.15275
Fooling LLM graders into giving better grades through neural activity guided adversarial prompting. (75%)
Atsushi Yamamura; Surya Ganguli
  The deployment of artificial intelligence (AI) in critical decision-making
and evaluation processes raises concerns about inherent biases that malicious
actors could exploit to distort decision outcomes. We propose a systematic
method to reveal such biases in AI evaluation systems and apply it to automated
essay grading as an example. Our approach first identifies hidden neural
activity patterns that predict distorted decision outcomes and then optimizes
an adversarial input suffix to amplify such patterns. We demonstrate that this
combination can effectively fool large language model (LLM) graders into
assigning much higher grades than humans would. We further show that this
white-box attack transfers to black-box attacks on other models, including
commercial closed-source models like Gemini. They further reveal the existence
of a "magic word" that plays a pivotal role in the efficacy of the attack. We
trace the origin of this magic word bias to the structure of commonly-used chat
templates for supervised fine-tuning of LLMs and show that a minor change in
the template can drastically reduce the bias. This work not only uncovers
vulnerabilities in current LLMs but also proposes a systematic method to
identify and remove hidden biases, contributing to the goal of ensuring AI
safety and security.


http://arxiv.org/abs/2412.13134
Practicable Black-box Evasion Attacks on Link Prediction in Dynamic Graphs -- A Graph Sequential Embedding Method. (70%)
Jiate Li; Meng Pang; Binghui Wang
  Link prediction in dynamic graphs (LPDG) has been widely applied to
real-world applications such as website recommendation, traffic flow
prediction, organizational studies, etc. These models are usually kept local
and secure, with only the interactive interface restrictively available to the
public. Thus, the problem of the black-box evasion attack on the LPDG model,
where model interactions and data perturbations are restricted, seems to be
essential and meaningful in practice. In this paper, we propose the first
practicable black-box evasion attack method that achieves effective attacks
against the target LPDG model, within a limited amount of interactions and
perturbations. To perform effective attacks under limited perturbations, we
develop a graph sequential embedding model to find the desired state embedding
of the dynamic graph sequences, under a deep reinforcement learning framework.
To overcome the scarcity of interactions, we design a multi-environment
training pipeline and train our agent for multiple instances, by sharing an
aggregate interaction buffer. Finally, we evaluate our attack against three
advanced LPDG models on three real-world graph datasets of different scales and
compare its performance with related methods under the interaction and
perturbation constraints. Experimental results show that our attack is both
effective and practicable.


http://arxiv.org/abs/2412.12621
Jailbreaking? One Step Is Enough! (64%)
Weixiong Zheng; Peijian Zeng; Yiwei Li; Hongyan Wu; Nankai Lin; Junhao Chen; Aimin Yang; Yongmei Zhou
  Large language models (LLMs) excel in various tasks but remain vulnerable to
jailbreak attacks, where adversaries manipulate prompts to generate harmful
outputs. Examining jailbreak prompts helps uncover the shortcomings of LLMs.
However, current jailbreak methods and the target model's defenses are engaged
in an independent and adversarial process, resulting in the need for frequent
attack iterations and redesigning attacks for different models. To address
these gaps, we propose a Reverse Embedded Defense Attack (REDA) mechanism that
disguises the attack intention as the "defense". intention against harmful
content. Specifically, REDA starts from the target response, guiding the model
to embed harmful content within its defensive measures, thereby relegating
harmful content to a secondary role and making the model believe it is
performing a defensive task. The attacking model considers that it is guiding
the target model to deal with harmful content, while the target model thinks it
is performing a defensive task, creating an illusion of cooperation between the
two. Additionally, to enhance the model's confidence and guidance in
"defensive" intentions, we adopt in-context learning (ICL) with a small number
of attack examples and construct a corresponding dataset of attack examples.
Extensive evaluations demonstrate that the REDA method enables cross-model
attacks without the need to redesign attack strategies for different models,
enables successful jailbreak in one iteration, and outperforms existing methods
on both open-source and closed-source models.


http://arxiv.org/abs/2412.15267
Toxicity Detection towards Adaptability to Changing Perturbations. (11%)
Hankun Kang; Jianhao Chen; Yongqi Li; Xin Miao; Mayi Xu; Ming Zhong; Yuanyuan Zhu; Tieyun Qian
  Toxicity detection is crucial for maintaining the peace of the society. While
existing methods perform well on normal toxic contents or those generated by
specific perturbation methods, they are vulnerable to evolving perturbation
patterns. However, in real-world scenarios, malicious users tend to create new
perturbation patterns for fooling the detectors. For example, some users may
circumvent the detector of large language models (LLMs) by adding `I am a
scientist' at the beginning of the prompt. In this paper, we introduce a novel
problem, i.e., continual learning jailbreak perturbation patterns, into the
toxicity detection field. To tackle this problem, we first construct a new
dataset generated by 9 types of perturbation patterns, 7 of them are summarized
from prior work and 2 of them are developed by us. We then systematically
validate the vulnerability of current methods on this new perturbation
pattern-aware dataset via both the zero-shot and fine tuned cross-pattern
detection. Upon this, we present the domain incremental learning paradigm and
the corresponding benchmark to ensure the detector's robustness to dynamically
emerging types of perturbed toxic text. Our code and dataset are provided in
the appendix and will be publicly available at GitHub, by which we wish to
offer new research opportunities for the security-relevant communities.


http://arxiv.org/abs/2412.13017
A New Adversarial Perspective for LiDAR-based 3D Object Detection. (9%)
Shijun Zheng; Weiquan Liu; Yu Guo; Yu Zang; Siqi Shen; Cheng Wang
  Autonomous vehicles (AVs) rely on LiDAR sensors for environmental perception
and decision-making in driving scenarios. However, ensuring the safety and
reliability of AVs in complex environments remains a pressing challenge. To
address this issue, we introduce a real-world dataset (ROLiD) comprising
LiDAR-scanned point clouds of two random objects: water mist and smoke. In this
paper, we introduce a novel adversarial perspective by proposing an attack
framework that utilizes water mist and smoke to simulate environmental
interference. Specifically, we propose a point cloud sequence generation method
using a motion and content decomposition generative adversarial network named
PCS-GAN to simulate the distribution of random objects. Furthermore, leveraging
the simulated LiDAR scanning characteristics implemented with Range Image, we
examine the effects of introducing random object perturbations at various
positions on the target vehicle. Extensive experiments demonstrate that
adversarial perturbations based on random objects effectively deceive vehicle
detection and reduce the recognition rate of 3D object detection models.


http://arxiv.org/abs/2412.13099
Accuracy Limits as a Barrier to Biometric System Security. (2%)
Axel Durbet; Paul-Marie Grollemund; Pascal Lafourcade; Kevin Thiry-Atighehchi
  Biometric systems are widely used for identity verification and
identification, including authentication (i.e., one-to-one matching to verify a
claimed identity) and identification (i.e., one-to-many matching to find a
subject in a database). The matching process relies on measuring similarities
or dissimilarities between a fresh biometric template and enrolled templates.
The False Match Rate FMR is a key metric for assessing the accuracy and
reliability of such systems. This paper analyzes biometric systems based on
their FMR, with two main contributions. First, we explore untargeted attacks,
where an adversary aims to impersonate any user within a database. We determine
the number of trials required for an attacker to successfully impersonate a
user and derive the critical population size (i.e., the maximum number of users
in the database) required to maintain a given level of security. Furthermore,
we compute the critical FMR value needed to ensure resistance against
untargeted attacks as the database size increases. Second, we revisit the
biometric birthday problem to evaluate the approximate and exact probabilities
that two users in a database collide (i.e., can impersonate each other). Based
on this analysis, we derive both the approximate critical population size and
the critical FMR value needed to bound the likelihood of such collisions
occurring with a given probability. These thresholds offer insights for
designing systems that mitigate the risk of impersonation and collisions,
particularly in large-scale biometric databases. Our findings indicate that
current biometric systems fail to deliver sufficient accuracy to achieve an
adequate security level against untargeted attacks, even in small-scale
databases. Moreover, state-of-the-art systems face significant challenges in
addressing the biometric birthday problem, especially as database sizes grow.


http://arxiv.org/abs/2412.12996
Neural Control and Certificate Repair via Runtime Monitoring. (1%)
Emily Yu; Đorđe Žikelić; Thomas A. Henzinger
  Learning-based methods provide a promising approach to solving highly
non-linear control tasks that are often challenging for classical control
methods. To ensure the satisfaction of a safety property, learning-based
methods jointly learn a control policy together with a certificate function for
the property. Popular examples include barrier functions for safety and
Lyapunov functions for asymptotic stability. While there has been significant
progress on learning-based control with certificate functions in the white-box
setting, where the correctness of the certificate function can be formally
verified, there has been little work on ensuring their reliability in the
black-box setting where the system dynamics are unknown. In this work, we
consider the problems of certifying and repairing neural network control
policies and certificate functions in the black-box setting. We propose a novel
framework that utilizes runtime monitoring to detect system behaviors that
violate the property of interest under some initially trained neural network
policy and certificate. These violating behaviors are used to extract new
training data, that is used to re-train the neural network policy and the
certificate function and to ultimately repair them. We demonstrate the
effectiveness of our approach empirically by using it to repair and to boost
the safety rate of neural network policies learned by a state-of-the-art method
for learning-based control on two autonomous system control tasks.


http://arxiv.org/abs/2412.13229
Training Verification-Friendly Neural Networks via Neuron Behavior Consistency. (1%)
Zongxin Liu; Zhe Zhao; Fu Song; Jun Sun; Pengfei Yang; Xiaowei Huang; Lijun Zhang
  Formal verification provides critical security assurances for neural
networks, yet its practical application suffers from the long verification
time. This work introduces a novel method for training verification-friendly
neural networks, which are robust, easy to verify, and relatively accurate. Our
method integrates neuron behavior consistency into the training process, making
neuron activation states consistent across different inputs in a local
neighborhood, reducing the number of unstable neurons and tightening the bounds
of neurons thereby enhancing neural network verifiability. We evaluated our
method using the MNIST, Fashion-MNIST, and CIFAR-10 datasets across various
network architectures. The results of the experiment demonstrate that networks
trained using our method are verification-friendly across different radii and
different model architectures, whereas other tools fail to maintain
verifiability as the radius increases. We also show that our method can be
combined with existing methods to further improve the verifiability of
networks.


http://arxiv.org/abs/2412.13394
Distribution Shifts at Scale: Out-of-distribution Detection in Earth Observation. (1%)
Burak Ekim; Girmaw Abebe Tadesse; Caleb Robinson; Gilles Hacheme; Michael Schmitt; Rahul Dodhia; Juan M. Lavista Ferres
  Training robust deep learning models is critical in Earth Observation, where
globally deployed models often face distribution shifts that degrade
performance, especially in low-data regions. Out-of-distribution (OOD)
detection addresses this challenge by identifying inputs that differ from
in-distribution (ID) data. However, existing methods either assume access to
OOD data or compromise primary task performance, making them unsuitable for
real-world deployment. We propose TARDIS, a post-hoc OOD detection method for
scalable geospatial deployments. The core novelty lies in generating surrogate
labels by integrating information from ID data and unknown distributions,
enabling OOD detection at scale. Our method takes a pre-trained model, ID data,
and WILD samples, disentangling the latter into surrogate ID and surrogate OOD
labels based on internal activations, and fits a binary classifier as an OOD
detector. We validate TARDIS on EuroSAT and xBD datasets, across 17
experimental setups covering covariate and semantic shifts, showing that it
performs close to the theoretical upper bound in assigning surrogate ID and OOD
samples in 13 cases. To demonstrate scalability, we deploy TARDIS on the Fields
of the World dataset, offering actionable insights into pre-trained model
behavior for large-scale deployments. The code is publicly available at
https://github.com/microsoft/geospatial-ood-detection.


http://arxiv.org/abs/2412.12449
Adversarially robust generalization theory via Jacobian regularization for deep neural networks. (99%)
Dongya Wu; Xin Li
  Powerful deep neural networks are vulnerable to adversarial attacks. To
obtain adversarially robust models, researchers have separately developed
adversarial training and Jacobian regularization techniques. There are abundant
theoretical and empirical studies for adversarial training, but theoretical
foundations for Jacobian regularization are still lacking. In this study, we
show that Jacobian regularization is closely related to adversarial training in
that $\ell_{2}$ or $\ell_{1}$ Jacobian regularized loss serves as an
approximate upper bound on the adversarially robust loss under $\ell_{2}$ or
$\ell_{\infty}$ adversarial attack respectively. Further, we establish the
robust generalization gap for Jacobian regularized risk minimizer via bounding
the Rademacher complexity of both the standard loss function class and Jacobian
regularization function class. Our theoretical results indicate that the norms
of Jacobian are related to both standard and robust generalization. We also
perform experiments on MNIST data classification to demonstrate that Jacobian
regularized risk minimization indeed serves as a surrogate for adversarially
robust risk minimization, and that reducing the norms of Jacobian can improve
both standard and robust generalization. This study promotes both theoretical
and empirical understandings to adversarially robust generalization via
Jacobian regularization.


http://arxiv.org/abs/2412.12478
Human-in-the-Loop Generation of Adversarial Texts: A Case Study on Tibetan Script. (99%)
Xi Cao; Yuan Sun; Jiajun Li; Quzong Gesang; Nuo Qun; Tashi Nyima
  DNN-based language models perform excellently on various tasks, but even SOTA
LLMs are susceptible to textual adversarial attacks. Adversarial texts play
crucial roles in multiple subfields of NLP. However, current research has the
following issues. (1) Most textual adversarial attack methods target
rich-resourced languages. How do we generate adversarial texts for less-studied
languages? (2) Most textual adversarial attack methods are prone to generating
invalid or ambiguous adversarial texts. How do we construct high-quality
adversarial robustness benchmarks? (3) New language models may be immune to
part of previously generated adversarial texts. How do we update adversarial
robustness benchmarks? To address the above issues, we introduce HITL-GAT, a
system based on a general approach to human-in-the-loop generation of
adversarial texts. HITL-GAT contains four stages in one pipeline: victim model
construction, adversarial example generation, high-quality benchmark
construction, and adversarial robustness evaluation. Additionally, we utilize
HITL-GAT to make a case study on Tibetan script which can be a reference for
the adversarial research of other less-studied languages.


http://arxiv.org/abs/2412.11735
Transferable Adversarial Face Attack with Text Controlled Attribute. (98%)
Wenyun Li; Zheng Zhang; Xiangyuan Lan; Dongmei Jiang
  Traditional adversarial attacks typically produce adversarial examples under
norm-constrained conditions, whereas unrestricted adversarial examples are
free-form with semantically meaningful perturbations. Current unrestricted
adversarial impersonation attacks exhibit limited control over adversarial face
attributes and often suffer from low transferability. In this paper, we propose
a novel Text Controlled Attribute Attack (TCA$^2$) to generate photorealistic
adversarial impersonation faces guided by natural language. Specifically, the
category-level personal softmax vector is employed to precisely guide the
impersonation attacks. Additionally, we propose both data and model
augmentation strategies to achieve transferable attacks on unknown target
models. Finally, a generative model, \textit{i.e}, Style-GAN, is utilized to
synthesize impersonated faces with desired attributes. Extensive experiments on
two high-resolution face recognition datasets validate that our TCA$^2$ method
can generate natural text-guided adversarial impersonation faces with high
transferability. We also evaluate our method on real-world face recognition
systems, \textit{i.e}, Face++ and Aliyun, further demonstrating the practical
potential of our approach.


http://arxiv.org/abs/2412.11608
Towards Adversarial Robustness of Model-Level Mixture-of-Experts Architectures for Semantic Segmentation. (86%)
Svetlana Pavlitska; Enrico Eisen; J. Marius Zöllner
  Vulnerability to adversarial attacks is a well-known deficiency of deep
neural networks. Larger networks are generally more robust, and ensembling is
one method to increase adversarial robustness: each model's weaknesses are
compensated by the strengths of others. While an ensemble uses a deterministic
rule to combine model outputs, a mixture of experts (MoE) includes an
additional learnable gating component that predicts weights for the outputs of
the expert models, thus determining their contributions to the final
prediction. MoEs have been shown to outperform ensembles on specific tasks, yet
their susceptibility to adversarial attacks has not been studied yet. In this
work, we evaluate the adversarial vulnerability of MoEs for semantic
segmentation of urban and highway traffic scenes. We show that MoEs are, in
most cases, more robust to per-instance and universal white-box adversarial
attacks and can better withstand transfer attacks. Our code is available at
\url{https://github.com/KASTEL-MobilityLab/mixtures-of-experts/}.


http://arxiv.org/abs/2412.11487
WFCAT: Augmenting Website Fingerprinting with Channel-wise Attention on Timing Features. (33%)
Jiajun Gong; Wei Cai; Siyuan Liang; Zhong Guan; Tao Wang; Ee-Chien Chang
  Website Fingerprinting (WF) aims to deanonymize users on the Tor network by
analyzing encrypted network traffic. Recent deep-learning-based attacks show
high accuracy on undefended traces. However, they struggle against modern
defenses that use tactics like injecting dummy packets and delaying real
packets, which significantly degrade classification performance. Our analysis
reveals that current attacks inadequately leverage the timing information
inherent in traffic traces, which persists as a source of leakage even under
robust defenses. Addressing this shortfall, we introduce a novel feature
representation named the Inter-Arrival Time (IAT) histogram, which quantifies
the frequencies of packet inter-arrival times across predetermined time slots.
Complementing this feature, we propose a new CNN-based attack, WFCAT, enhanced
with two innovative architectural blocks designed to optimally extract and
utilize timing information. Our approach uses kernels of varying sizes to
capture multi-scale features, which are then integrated using a weighted sum
across all feature channels to enhance the model's efficacy in identifying
temporal patterns. Our experiments validate that WFCAT substantially
outperforms existing methods on defended traces in both closed- and open-world
scenarios. Notably, WFCAT achieves over 59% accuracy against Surakav, a
recently developed robust defense, marking an improvement of over 28% and 48%
against the state-of-the-art attacks RF and Tik-Tok, respectively, in the
closed-world scenario.


http://arxiv.org/abs/2412.11471
Red Pill and Blue Pill: Controllable Website Fingerprinting Defense via Dynamic Backdoor Learning. (22%)
Siyuan Liang; Jiajun Gong; Tianmeng Fang; Aishan Liu; Tao Wang; Xianglong Liu; Xiaochun Cao; Dacheng Tao; Chang Ee-Chien
  Website fingerprint (WF) attacks, which covertly monitor user communications
to identify the web pages they visit, pose a serious threat to user privacy.
Existing WF defenses attempt to reduce the attacker's accuracy by disrupting
unique traffic patterns; however, they often suffer from the trade-off between
overhead and effectiveness, resulting in less usefulness in practice. To
overcome this limitation, we introduce Controllable Website Fingerprint Defense
(CWFD), a novel defense perspective based on backdoor learning. CWFD exploits
backdoor vulnerabilities in neural networks to directly control the attacker's
model by designing trigger patterns based on network traffic. Specifically,
CWFD injects only incoming packets on the server side into the target web
page's traffic, keeping overhead low while effectively poisoning the attacker's
model during training. During inference, the defender can influence the
attacker's model through a 'red pill, blue pill' choice: traces with the
trigger (red pill) lead to misclassification as the target web page, while
normal traces (blue pill) are classified correctly, achieving directed control
over the defense outcome. We use the Fast Levenshtein-like distance as the
optimization objective to compute trigger patterns that can be effectively
associated with our target page. Experiments show that CWFD significantly
reduces RF's accuracy from 99% to 6% with 74% data overhead. In comparison,
FRONT reduces accuracy to only 97% at similar overhead, while Palette achieves
32% accuracy with 48% more overhead. We further validate the practicality of
our method in a real Tor network environment.


http://arxiv.org/abs/2412.11840
Sonar-based Deep Learning in Underwater Robotics: Overview, Robustness and Challenges. (1%)
Martin Aubard; Ana Madureira; Luís Teixeira; José Pinto
  With the growing interest in underwater exploration and monitoring,
Autonomous Underwater Vehicles (AUVs) have become essential. The recent
interest in onboard Deep Learning (DL) has advanced real-time environmental
interaction capabilities relying on efficient and accurate vision-based DL
models. However, the predominant use of sonar in underwater environments,
characterized by limited training data and inherent noise, poses challenges to
model robustness. This autonomy improvement raises safety concerns for
deploying such models during underwater operations, potentially leading to
hazardous situations. This paper aims to provide the first comprehensive
overview of sonar-based DL under the scope of robustness. It studies
sonar-based DL perception task models, such as classification, object
detection, segmentation, and SLAM. Furthermore, the paper systematizes
sonar-based state-of-the-art datasets, simulators, and robustness methods such
as neural network verification, out-of-distribution, and adversarial attacks.
This paper highlights the lack of robustness in sonar-based DL research and
suggests future research pathways, notably establishing a baseline sonar-based
dataset and bridging the simulation-to-reality gap.


http://arxiv.org/abs/2412.12217
Comprehensive Survey on Adversarial Examples in Cybersecurity: Impacts, Challenges, and Mitigation Strategies. (99%)
Li Li
  Deep learning (DL) has significantly transformed cybersecurity, enabling
advancements in malware detection, botnet identification, intrusion detection,
user authentication, and encrypted traffic analysis. However, the rise of
adversarial examples (AE) poses a critical challenge to the robustness and
reliability of DL-based systems. These subtle, crafted perturbations can
deceive models, leading to severe consequences like misclassification and
system vulnerabilities. This paper provides a comprehensive review of the
impact of AE attacks on key cybersecurity applications, highlighting both their
theoretical and practical implications. We systematically examine the methods
used to generate adversarial examples, their specific effects across various
domains, and the inherent trade-offs attackers face between efficacy and
resource efficiency. Additionally, we explore recent advancements in defense
mechanisms, including gradient masking, adversarial training, and detection
techniques, evaluating their potential to enhance model resilience. By
summarizing cutting-edge research, this study aims to bridge the gap between
adversarial research and practical security applications, offering insights to
fortify the adoption of DL solutions in cybersecurity.


http://arxiv.org/abs/2412.11172
Unpacking the Resilience of SNLI Contradiction Examples to Attacks. (99%)
Chetan Verma; Archit Agarwal
  Pre-trained models excel on NLI benchmarks like SNLI and MultiNLI, but their
true language understanding remains uncertain. Models trained only on
hypotheses and labels achieve high accuracy, indicating reliance on dataset
biases and spurious correlations. To explore this issue, we applied the
Universal Adversarial Attack to examine the model's vulnerabilities. Our
analysis revealed substantial drops in accuracy for the entailment and neutral
classes, whereas the contradiction class exhibited a smaller decline.
Fine-tuning the model on an augmented dataset with adversarial examples
restored its performance to near-baseline levels for both the standard and
challenge sets. Our findings highlight the value of adversarial triggers in
identifying spurious correlations and improving robustness while providing
insights into the resilience of the contradiction class to adversarial attacks.


http://arxiv.org/abs/2412.11119
Impact of Adversarial Attacks on Deep Learning Model Explainability. (99%)
Gazi Nazia Nur; Mohammad Ahnaf Sadat
  In this paper, we investigate the impact of adversarial attacks on the
explainability of deep learning models, which are commonly criticized for their
black-box nature despite their capacity for autonomous feature extraction. This
black-box nature can affect the perceived trustworthiness of these models. To
address this, explainability techniques such as GradCAM, SmoothGrad, and LIME
have been developed to clarify model decision-making processes. Our research
focuses on the robustness of these explanations when models are subjected to
adversarial attacks, specifically those involving subtle image perturbations
that are imperceptible to humans but can significantly mislead models. For
this, we utilize attack methods like the Fast Gradient Sign Method (FGSM) and
the Basic Iterative Method (BIM) and observe their effects on model accuracy
and explanations. The results reveal a substantial decline in model accuracy,
with accuracies dropping from 89.94% to 58.73% and 45.50% under FGSM and BIM
attacks, respectively. Despite these declines in accuracy, the explanation of
the models measured by metrics such as Intersection over Union (IoU) and Root
Mean Square Error (RMSE) shows negligible changes, suggesting that these
metrics may not be sensitive enough to detect the presence of adversarial
perturbations.


http://arxiv.org/abs/2412.11441
UIBDiffusion: Universal Imperceptible Backdoor Attack for Diffusion Models. (99%)
Yuning Han; Bingyin Zhao; Rui Chu; Feng Luo; Biplab Sikdar; Yingjie Lao
  Recent studies show that diffusion models (DMs) are vulnerable to backdoor
attacks. Existing backdoor attacks impose unconcealed triggers (e.g., a gray
box and eyeglasses) that contain evident patterns, rendering remarkable attack
effects yet easy detection upon human inspection and defensive algorithms.
While it is possible to improve stealthiness by reducing the strength of the
backdoor, doing so can significantly compromise its generality and
effectiveness. In this paper, we propose UIBDiffusion, the universal
imperceptible backdoor attack for diffusion models, which allows us to achieve
superior attack and generation performance while evading state-of-the-art
defenses. We propose a novel trigger generation approach based on universal
adversarial perturbations (UAPs) and reveal that such perturbations, which are
initially devised for fooling pre-trained discriminative models, can be adapted
as potent imperceptible backdoor triggers for DMs. We evaluate UIBDiffusion on
multiple types of DMs with different kinds of samplers across various datasets
and targets. Experimental results demonstrate that UIBDiffusion brings three
advantages: 1) Universality, the imperceptible trigger is universal (i.e.,
image and model agnostic) where a single trigger is effective to any images and
all diffusion models with different samplers; 2) Utility, it achieves
comparable generation quality (e.g., FID) and even better attack success rate
(i.e., ASR) at low poison rates compared to the prior works; and 3)
Undetectability, UIBDiffusion is plausible to human perception and can bypass
Elijah and TERD, the SOTA defenses against backdoors for DMs. We will release
our backdoor triggers and code.


http://arxiv.org/abs/2412.11168
PGD-Imp: Rethinking and Unleashing Potential of Classic PGD with Dual Strategies for Imperceptible Adversarial Attacks. (98%)
Jin Li; Zitong Yu; Ziqiang He; Z. Jane Wang; Xiangui Kang
  Imperceptible adversarial attacks have recently attracted increasing research
interests. Existing methods typically incorporate external modules or loss
terms other than a simple $l_p$-norm into the attack process to achieve
imperceptibility, while we argue that such additional designs may not be
necessary. In this paper, we rethink the essence of imperceptible attacks and
propose two simple yet effective strategies to unleash the potential of PGD,
the common and classical attack, for imperceptibility from an optimization
perspective. Specifically, the Dynamic Step Size is introduced to find the
optimal solution with minimal attack cost towards the decision boundary of the
attacked model, and the Adaptive Early Stop strategy is adopted to reduce the
redundant strength of adversarial perturbations to the minimum level. The
proposed PGD-Imperceptible (PGD-Imp) attack achieves state-of-the-art results
in imperceptible adversarial attacks for both untargeted and targeted
scenarios. When performing untargeted attacks against ResNet-50, PGD-Imp
attains 100$\%$ (+0.3$\%$) ASR, 0.89 (-1.76) $l_2$ distance, and 52.93 (+9.2)
PSNR with 57s (-371s) running time, significantly outperforming existing
methods.


http://arxiv.org/abs/2412.11066
Learning Robust and Privacy-Preserving Representations via Information Theory. (92%)
Binghui Zhang; Sayedeh Leila Noorbakhsh; Yun Dong; Yuan Hong; Binghui Wang
  Machine learning models are vulnerable to both security attacks (e.g.,
adversarial examples) and privacy attacks (e.g., private attribute inference).
We take the first step to mitigate both the security and privacy attacks, and
maintain task utility as well. Particularly, we propose an
information-theoretic framework to achieve the goals through the lens of
representation learning, i.e., learning representations that are robust to both
adversarial examples and attribute inference adversaries. We also derive novel
theoretical results under our framework, e.g., the inherent trade-off between
adversarial robustness/utility and attribute privacy, and guaranteed attribute
privacy leakage against attribute inference adversaries.


http://arxiv.org/abs/2412.11384
A Comprehensive Review of Adversarial Attacks on Machine Learning. (75%)
Syed Quiser Ahmed; Bharathi Vokkaliga Ganesh; Sathyanarayana Sampath Kumar; Prakhar Mishra; Ravi Anand; Bhanuteja Akurathi
  This research provides a comprehensive overview of adversarial attacks on AI
and ML models, exploring various attack types, techniques, and their potential
harms. We also delve into the business implications, mitigation strategies, and
future research directions. To gain practical insights, we employ the
Adversarial Robustness Toolbox (ART) [1] library to simulate these attacks on
real-world use cases, such as self-driving cars. Our goal is to inform
practitioners and researchers about the challenges and opportunities in
defending AI systems against adversarial threats. By providing a comprehensive
comparison of different attack methods, we aim to contribute to the development
of more robust and secure AI systems.


http://arxiv.org/abs/2412.12212
Finding a Wolf in Sheep's Clothing: Combating Adversarial Text-To-Image Prompts with Text Summarization. (2%)
Portia Cooper; Harshita Narnoli; Mihai Surdeanu
  Text-to-image models are vulnerable to the stepwise "Divide-and-Conquer
Attack" (DACA) that utilize a large language model to obfuscate inappropriate
content in prompts by wrapping sensitive text in a benign narrative. To
mitigate stepwise DACA attacks, we propose a two-layer method involving text
summarization followed by binary classification. We assembled the Adversarial
Text-to-Image Prompt (ATTIP) dataset ($N=940$), which contained DACA-obfuscated
and non-obfuscated prompts. From the ATTIP dataset, we created two summarized
versions: one generated by a small encoder model and the other by a large
language model. Then, we used an encoder classifier and a GPT-4o classifier to
perform content moderation on the summarized and unsummarized prompts. When
compared with a classifier that operated over the unsummarized data, our method
improved F1 score performance by 31%. Further, the highest recorded F1 score
achieved (98%) was produced by the encoder classifier on a summarized ATTIP
variant. This study indicates that pre-classification text summarization can
inoculate content detection models against stepwise DACA obfuscations.


http://arxiv.org/abs/2412.11057
Set-Valued Sensitivity Analysis of Deep Neural Networks. (2%)
Xin Jeff Wang; Feiling Jeff wang; Xuegang Jeff Ban
  This paper proposes a sensitivity analysis framework based on set valued
mapping for deep neural networks (DNN) to understand and compute how the
solutions (model weights) of DNN respond to perturbations in the training data.
As a DNN may not exhibit a unique solution (minima) and the algorithm of
solving a DNN may lead to different solutions with minor perturbations to input
data, we focus on the sensitivity of the solution set of DNN, instead of
studying a single solution. In particular, we are interested in the expansion
and contraction of the set in response to data perturbations. If the change of
solution set can be bounded by the extent of the data perturbation, the model
is said to exhibit the Lipschitz like property. This "set-to-set" analysis
approach provides a deeper understanding of the robustness and reliability of
DNNs during training. Our framework incorporates both isolated and non-isolated
minima, and critically, does not require the assumption that the Hessian of
loss function is non-singular. By developing set-level metrics such as distance
between sets, convergence of sets, derivatives of set-valued mapping, and
stability across the solution set, we prove that the solution set of the Fully
Connected Neural Network holds Lipschitz-like properties. For general neural
networks (e.g., Resnet), we introduce a graphical-derivative-based method to
estimate the new solution set following data perturbation without retraining.


http://arxiv.org/abs/2412.11390
Accurate, Robust and Privacy-Preserving Brain-Computer Interface Decoding. (1%)
Xiaoqing Chen; Tianwang Jia; Dongrui Wu
  An electroencephalogram (EEG) based brain-computer interface (BCI) enables
direct communication between the brain and external devices. However, EEG-based
BCIs face at least three major challenges in real-world applications: data
scarcity and individual differences, adversarial vulnerability, and data
privacy. While previous studies have addressed one or two of these issues,
simultaneous accommodation of all three challenges remains challenging and
unexplored. This paper fills this gap, by proposing an Augmented Robustness
Ensemble (ARE) algorithm and integrating it into three privacy protection
scenarios (centralized source-free transfer, federated source-free transfer,
and source data perturbation), achieving simultaneously accurate decoding,
adversarial robustness, and privacy protection of EEG-based BCIs. Experiments
on three public EEG datasets demonstrated that our proposed approach
outperformed over 10 classic and state-of-the-art approaches in both accuracy
and robustness in all three privacy-preserving scenarios, even outperforming
state-of-the-art transfer learning approaches that do not consider privacy
protection at all. This is the first time that three major challenges in
EEG-based BCIs can be addressed simultaneously, significantly improving the
practicalness of EEG decoding in real-world BCIs.


http://arxiv.org/abs/2412.11109
SpearBot: Leveraging Large Language Models in a Generative-Critique Framework for Spear-Phishing Email Generation. (1%)
Qinglin Qi; Yun Luo; Yijia Xu; Wenbo Guo; Yong Fang
  Large Language Models (LLMs) are increasingly capable, aiding in tasks such
as content generation, yet they also pose risks, particularly in generating
harmful spear-phishing emails. These emails, crafted to entice clicks on
malicious URLs, threaten personal information security. This paper proposes an
adversarial framework, SpearBot, which utilizes LLMs to generate spear-phishing
emails with various phishing strategies. Through specifically crafted jailbreak
prompts, SpearBot circumvents security policies and introduces other LLM
instances as critics. When a phishing email is identified by the critic,
SpearBot refines the generated email based on the critique feedback until it
can no longer be recognized as phishing, thereby enhancing its deceptive
quality. To evaluate the effectiveness of SpearBot, we implement various
machine-based defenders and assess how well the phishing emails generated could
deceive them. Results show these emails often evade detection to a large
extent, underscoring their deceptive quality. Additionally, human evaluations
of the emails' readability and deception are conducted through questionnaires,
confirming their convincing nature and the significant potential harm of the
generated phishing emails.


http://arxiv.org/abs/2412.10713
RAT: Adversarial Attacks on Deep Reinforcement Agents for Targeted Behaviors. (98%)
Fengshuo Bai; Runze Liu; Yali Du; Ying Wen; Yaodong Yang
  Evaluating deep reinforcement learning (DRL) agents against targeted behavior
attacks is critical for assessing their robustness. These attacks aim to
manipulate the victim into specific behaviors that align with the attacker's
objectives, often bypassing traditional reward-based defenses. Prior methods
have primarily focused on reducing cumulative rewards; however, rewards are
typically too generic to capture complex safety requirements effectively. As a
result, focusing solely on reward reduction can lead to suboptimal attack
strategies, particularly in safety-critical scenarios where more precise
behavior manipulation is needed. To address these challenges, we propose RAT, a
method designed for universal, targeted behavior attacks. RAT trains an
intention policy that is explicitly aligned with human preferences, serving as
a precise behavioral target for the adversary. Concurrently, an adversary
manipulates the victim's policy to follow this target behavior. To enhance the
effectiveness of these attacks, RAT dynamically adjusts the state occupancy
measure within the replay buffer, allowing for more controlled and effective
behavior manipulation. Our empirical results on robotic simulation tasks
demonstrate that RAT outperforms existing adversarial attack algorithms in
inducing specific behaviors. Additionally, RAT shows promise in improving agent
robustness, leading to more resilient policies. We further validate RAT by
guiding Decision Transformer agents to adopt behaviors aligned with human
preferences in various MuJoCo tasks, demonstrating its effectiveness across
diverse tasks.


http://arxiv.org/abs/2412.10681
One Pixel is All I Need. (80%)
Deng Siqin; Zhou Xiaoyi
  Vision Transformers (ViTs) have achieved record-breaking performance in
various visual tasks. However, concerns about their robustness against backdoor
attacks have grown. Backdoor attacks involve associating a specific trigger
with a target label, causing the model to predict the attacker-specified label
when the trigger is present, while correctly identifying clean images.We found
that ViTs exhibit higher attack success rates for quasi-triggers(patterns
different from but similar to the original training triggers)compared to CNNs.
Moreover, some backdoor features in clean samples can suppress the original
trigger, making quasi-triggers more effective.To better understand and exploit
these vulnerabilities, we developed a tool called the Perturbation Sensitivity
Distribution Map (PSDM). PSDM computes and sums gradients over many inputs to
show how sensitive the model is to small changes in the input. In ViTs, PSDM
reveals a patch-like pattern where central pixels are more sensitive than
edges. We use PSDM to guide the creation of quasi-triggers.Based on these
findings, we designed "WorstVIT," a simple yet effective data poisoning
backdoor for ViT models. This attack requires an extremely low poisoning rate,
trains for just one epoch, and modifies a single pixel to successfully attack
all validation images.


http://arxiv.org/abs/2412.10805
Are Language Models Agnostic to Linguistically Grounded Perturbations? A Case Study of Indic Languages. (1%)
Poulami Ghosh; Raj Dabre; Pushpak Bhattacharyya
  Pre-trained language models (PLMs) are known to be susceptible to
perturbations to the input text, but existing works do not explicitly focus on
linguistically grounded attacks, which are subtle and more prevalent in nature.
In this paper, we study whether PLMs are agnostic to linguistically grounded
attacks or not. To this end, we offer the first study addressing this,
investigating different Indic languages and various downstream tasks. Our
findings reveal that although PLMs are susceptible to linguistic perturbations,
when compared to non-linguistic attacks, PLMs exhibit a slightly lower
susceptibility to linguistic attacks. This highlights that even constrained
attacks are effective. Moreover, we investigate the implications of these
outcomes across a range of languages, encompassing diverse language families
and different scripts.


http://arxiv.org/abs/2412.10831
Unbiased General Annotated Dataset Generation. (1%)
Dengyang Jiang; Haoyu Wang; Lei Zhang; Wei Wei; Guang Dai; Mengmeng Wang; Jingdong Wang; Yanning Zhang
  Pre-training backbone networks on a general annotated dataset (e.g.,
ImageNet) that comprises numerous manually collected images with category
annotations has proven to be indispensable for enhancing the generalization
capacity of downstream visual tasks. However, those manually collected images
often exhibit bias, which is non-transferable across either categories or
domains, thus causing the model's generalization capacity degeneration. To
mitigate this problem, we present an unbiased general annotated dataset
generation framework (ubGen). Instead of expensive manual collection, we aim at
directly generating unbiased images with category annotations. To achieve this
goal, we propose to leverage the advantage of a multimodal foundation model
(e.g., CLIP), in terms of aligning images in an unbiased semantic space defined
by language. Specifically, we develop a bi-level semantic alignment loss, which
not only forces all generated images to be consistent with the semantic
distribution of all categories belonging to the target dataset in an
adversarial learning manner, but also requires each generated image to match
the semantic description of its category name. In addition, we further cast an
existing image quality scoring model into a quality assurance loss to preserve
the quality of the generated image. By leveraging these two loss functions, we
can obtain an unbiased image generation model by simply fine-tuning a
pre-trained diffusion model using only all category names in the target dataset
as input. Experimental results confirm that, compared with the manually labeled
dataset or other synthetic datasets, the utilization of our generated unbiased
datasets leads to stable generalization capacity enhancement of different
backbone networks across various tasks, especially in tasks where the manually
labeled samples are scarce.


http://arxiv.org/abs/2412.10353
Robust image classification with multi-modal large language models. (99%)
Francesco Villani; Igor Maljkovic; Dario Lazzaro; Angelo Sotgiu; Antonio Emanuele Cinà; Fabio Roli
  Deep Neural Networks are vulnerable to adversarial examples, i.e., carefully
crafted input samples that can cause models to make incorrect predictions with
high confidence. To mitigate these vulnerabilities, adversarial training and
detection-based defenses have been proposed to strengthen models in advance.
However, most of these approaches focus on a single data modality, overlooking
the relationships between visual patterns and textual descriptions of the
input. In this paper, we propose a novel defense, Multi-Shield, designed to
combine and complement these defenses with multi-modal information to further
enhance their robustness. Multi-Shield leverages multi-modal large language
models to detect adversarial examples and abstain from uncertain
classifications when there is no alignment between textual and visual
representations of the input. Extensive evaluations on CIFAR-10 and ImageNet
datasets, using robust and non-robust image classification models, demonstrate
that Multi-Shield can be easily integrated to detect and reject adversarial
examples, outperforming the original defenses.


http://arxiv.org/abs/2412.09910
Prompt2Perturb (P2P): Text-Guided Diffusion-Based Adversarial Attacks on Breast Ultrasound Images. (99%)
Yasamin Medghalchi; Moein Heidari; Clayton Allard; Leonid Sigal; Ilker Hacihaliloglu
  Deep neural networks (DNNs) offer significant promise for improving breast
cancer diagnosis in medical imaging. However, these models are highly
susceptible to adversarial attacks--small, imperceptible changes that can
mislead classifiers--raising critical concerns about their reliability and
security. Traditional attacks rely on fixed-norm perturbations, misaligning
with human perception. In contrast, diffusion-based attacks require pre-trained
models, demanding substantial data when these models are unavailable, limiting
practical use in data-scarce scenarios. In medical imaging, however, this is
often unfeasible due to the limited availability of datasets. Building on
recent advancements in learnable prompts, we propose Prompt2Perturb (P2P), a
novel language-guided attack method capable of generating meaningful attack
examples driven by text instructions. During the prompt learning phase, our
approach leverages learnable prompts within the text encoder to create subtle,
yet impactful, perturbations that remain imperceptible while guiding the model
towards targeted outcomes. In contrast to current prompt learning-based
approaches, our P2P stands out by directly updating text embeddings, avoiding
the need for retraining diffusion models. Further, we leverage the finding that
optimizing only the early reverse diffusion steps boosts efficiency while
ensuring that the generated adversarial examples incorporate subtle noise, thus
preserving ultrasound image quality without introducing noticeable artifacts.
We show that our method outperforms state-of-the-art attack techniques across
three breast ultrasound datasets in FID and LPIPS. Moreover, the generated
images are both more natural in appearance and more effective compared to
existing adversarial attacks. Our code will be publicly available
https://github.com/yasamin-med/P2P.


http://arxiv.org/abs/2412.09954
A2RNet: Adversarial Attack Resilient Network for Robust Infrared and Visible Image Fusion. (92%)
Jiawei Li; Hongwei Yu; Jiansheng Chen; Xinlong Ding; Jinlong Wang; Jinyuan Liu; Bochao Zou; Huimin Ma
  Infrared and visible image fusion (IVIF) is a crucial technique for enhancing
visual performance by integrating unique information from different modalities
into one fused image. Exiting methods pay more attention to conducting fusion
with undisturbed data, while overlooking the impact of deliberate interference
on the effectiveness of fusion results. To investigate the robustness of fusion
models, in this paper, we propose a novel adversarial attack resilient network,
called $\textrm{A}^{\textrm{2}}$RNet. Specifically, we develop an adversarial
paradigm with an anti-attack loss function to implement adversarial attacks and
training. It is constructed based on the intrinsic nature of IVIF and provide a
robust foundation for future research advancements. We adopt a Unet as the
pipeline with a transformer-based defensive refinement module (DRM) under this
paradigm, which guarantees fused image quality in a robust coarse-to-fine
manner. Compared to previous works, our method mitigates the adverse effects of
adversarial perturbations, consistently maintaining high-fidelity fusion
results. Furthermore, the performance of downstream tasks can also be well
maintained under adversarial attacks. Code is available at
https://github.com/lok-18/A2RNet.


http://arxiv.org/abs/2412.10597
Err on the Side of Texture: Texture Bias on Real Data. (82%)
Blaine Hoak; Ryan Sheatsley; Patrick McDaniel
  Bias significantly undermines both the accuracy and trustworthiness of
machine learning models. To date, one of the strongest biases observed in image
classification models is texture bias-where models overly rely on texture
information rather than shape information. Yet, existing approaches for
measuring and mitigating texture bias have not been able to capture how
textures impact model robustness in real-world settings. In this work, we
introduce the Texture Association Value (TAV), a novel metric that quantifies
how strongly models rely on the presence of specific textures when classifying
objects. Leveraging TAV, we demonstrate that model accuracy and robustness are
heavily influenced by texture. Our results show that texture bias explains the
existence of natural adversarial examples, where over 90% of these samples
contain textures that are misaligned with the learned texture of their true
label, resulting in confident mispredictions.


http://arxiv.org/abs/2412.10617
BinarySelect to Improve Accessibility of Black-Box Attack Research. (80%)
Shatarupa Ghosh; Jonathan Rusert
  Adversarial text attack research is useful for testing the robustness of NLP
models, however, the rise of transformers has greatly increased the time
required to test attacks. Especially when researchers do not have access to
adequate resources (e.g. GPUs). This can hinder attack research, as modifying
one example for an attack can require hundreds of queries to a model,
especially for black-box attacks. Often these attacks remove one token at a
time to find the ideal one to change, requiring $n$ queries (the length of the
text) right away. We propose a more efficient selection method called
BinarySelect which combines binary search and attack selection methods to
greatly reduce the number of queries needed to find a token. We find that
BinarySelect only needs $\text{log}_2(n) * 2$ queries to find the first token
compared to $n$ queries. We also test BinarySelect in an attack setting against
5 classifiers across 3 datasets and find a viable tradeoff between number of
queries saved and attack effectiveness. For example, on the Yelp dataset, the
number of queries is reduced by 32% (72 less) with a drop in attack
effectiveness of only 5 points. We believe that BinarySelect can help future
researchers study adversarial attacks and black-box problems more efficiently
and opens the door for researchers with access to less resources.


http://arxiv.org/abs/2412.10535
On Adversarial Robustness and Out-of-Distribution Robustness of Large Language Models. (78%)
April Yang; Jordan Tab; Parth Shah; Paul Kotchavong
  The increasing reliance on large language models (LLMs) for diverse
applications necessitates a thorough understanding of their robustness to
adversarial perturbations and out-of-distribution (OOD) inputs. In this study,
we investigate the correlation between adversarial robustness and OOD
robustness in LLMs, addressing a critical gap in robustness evaluation. By
applying methods originally designed to improve one robustness type across both
contexts, we analyze their performance on adversarial and out-of-distribution
benchmark datasets. The input of the model consists of text samples, with the
output prediction evaluated in terms of accuracy, precision, recall, and F1
scores in various natural language inference tasks.
  Our findings highlight nuanced interactions between adversarial robustness
and OOD robustness, with results indicating limited transferability between the
two robustness types. Through targeted ablations, we evaluate how these
correlations evolve with different model sizes and architectures, uncovering
model-specific trends: smaller models like LLaMA2-7b exhibit neutral
correlations, larger models like LLaMA2-13b show negative correlations, and
Mixtral demonstrates positive correlations, potentially due to domain-specific
alignment. These results underscore the importance of hybrid robustness
frameworks that integrate adversarial and OOD strategies tailored to specific
models and domains. Further research is needed to evaluate these interactions
across larger models and varied architectures, offering a pathway to more
reliable and generalizable LLMs.


http://arxiv.org/abs/2412.09921
FaceShield: Defending Facial Image against Deepfake Threats. (70%)
Jaehwan Jeong; Sumin In; Sieun Kim; Hannie Shin; Jongheon Jeong; Sang Ho Yoon; Jaewook Chung; Sangpil Kim
  The rising use of deepfakes in criminal activities presents a significant
issue, inciting widespread controversy. While numerous studies have tackled
this problem, most primarily focus on deepfake detection. These reactive
solutions are insufficient as a fundamental approach for crimes where
authenticity verification is not critical. Existing proactive defenses also
have limitations, as they are effective only for deepfake models based on
specific Generative Adversarial Networks (GANs), making them less applicable in
light of recent advancements in diffusion-based models. In this paper, we
propose a proactive defense method named FaceShield, which introduces novel
attack strategies targeting deepfakes generated by Diffusion Models (DMs) and
facilitates attacks on various existing GAN-based deepfake models through
facial feature extractor manipulations. Our approach consists of three main
components: (i) manipulating the attention mechanism of DMs to exclude
protected facial features during the denoising process, (ii) targeting
prominent facial feature extraction models to enhance the robustness of our
adversarial perturbation, and (iii) employing Gaussian blur and low-pass
filtering techniques to improve imperceptibility while enhancing robustness
against JPEG distortion. Experimental results on the CelebA-HQ and VGGFace2-HQ
datasets demonstrate that our method achieves state-of-the-art performance
against the latest deepfake models based on DMs, while also exhibiting
applicability to GANs and showcasing greater imperceptibility of noise along
with enhanced robustness.


http://arxiv.org/abs/2412.10049
SuperMark: Robust and Training-free Image Watermarking via Diffusion-based Super-Resolution. (67%)
Runyi Hu; Jie Zhang; Yiming Li; Jiwei Li; Qing Guo; Han Qiu; Tianwei Zhang
  In today's digital landscape, the blending of AI-generated and authentic
content has underscored the need for copyright protection and content
authentication. Watermarking has become a vital tool to address these
challenges, safeguarding both generated and real content. Effective
watermarking methods must withstand various distortions and attacks. Current
deep watermarking techniques often use an encoder-noise layer-decoder
architecture and include distortions to enhance robustness. However, they
struggle to balance robustness and fidelity and remain vulnerable to adaptive
attacks, despite extensive training. To overcome these limitations, we propose
SuperMark, a robust, training-free watermarking framework. Inspired by the
parallels between watermark embedding/extraction in watermarking and the
denoising/noising processes in diffusion models, SuperMark embeds the watermark
into initial Gaussian noise using existing techniques. It then applies
pre-trained Super-Resolution (SR) models to denoise the watermarked noise,
producing the final watermarked image. For extraction, the process is reversed:
the watermarked image is inverted back to the initial watermarked noise via
DDIM Inversion, from which the embedded watermark is extracted. This flexible
framework supports various noise injection methods and diffusion-based SR
models, enabling enhanced customization. The robustness of the DDIM Inversion
process against perturbations allows SuperMark to achieve strong resilience to
distortions while maintaining high fidelity. Experiments demonstrate that
SuperMark achieves fidelity comparable to existing methods while significantly
improving robustness. Under standard distortions, it achieves an average
watermark extraction accuracy of 99.46%, and 89.29% under adaptive attacks.
Moreover, SuperMark shows strong transferability across datasets, SR models,
embedding methods, and resolutions.


http://arxiv.org/abs/2412.10605
Client-Side Patching against Backdoor Attacks in Federated Learning. (61%)
Borja Molina-Coronado
  Federated learning is a versatile framework for training models in
decentralized environments. However, the trust placed in clients makes
federated learning vulnerable to backdoor attacks launched by malicious
participants. While many defenses have been proposed, they often fail short
when facing heterogeneous data distributions among participating clients. In
this paper, we propose a novel defense mechanism for federated learning systems
designed to mitigate backdoor attacks on the clients-side. Our approach
leverages adversarial learning techniques and model patching to neutralize the
impact of backdoor attacks. Through extensive experiments on the MNIST and
Fashion-MNIST datasets, we demonstrate that our defense effectively reduces
backdoor accuracy, outperforming existing state-of-the-art defenses, such as
LFighter, FLAME, and RoseAgg, in i.i.d. and non-i.i.d. scenarios, while
maintaining competitive or superior accuracy on clean data.


http://arxiv.org/abs/2412.12192
No Free Lunch for Defending Against Prefilling Attack by In-Context Learning. (22%)
Zhiyu Xue; Guangliang Liu; Bocheng Chen; Kristen Marie Johnson; Ramtin Pedarsani
  The security of Large Language Models (LLMs) has become an important research
topic since the emergence of ChatGPT. Though there have been various effective
methods to defend against jailbreak attacks, prefilling attacks remain an
unsolved and popular threat against open-sourced LLMs. In-Context Learning
(ICL) offers a computationally efficient defense against various jailbreak
attacks, yet no effective ICL methods have been developed to counter prefilling
attacks. In this paper, we: (1) show that ICL can effectively defend against
prefilling jailbreak attacks by employing adversative sentence structures
within demonstrations; (2) characterize the effectiveness of this defense
through the lens of model size, number of demonstrations, over-defense,
integration with other jailbreak attacks, and the presence of safety alignment.
Given the experimental results and our analysis, we conclude that there is no
free lunch for defending against prefilling jailbreak attacks with ICL. On the
one hand, current safety alignment methods fail to mitigate prefilling
jailbreak attacks, but adversative structures within ICL demonstrations provide
robust defense across various model sizes and complex jailbreak attacks. On the
other hand, LLMs exhibit similar over-defensiveness when utilizing ICL
demonstrations with adversative structures, and this behavior appears to be
independent of model size.


http://arxiv.org/abs/2412.09933
Active Poisoning: Efficient Backdoor Attacks on Transfer Learning-Based Brain-Computer Interfaces. (13%)
X. Jiang; L. Meng; S. Li; D. Wu
  Transfer learning (TL) has been widely used in electroencephalogram
(EEG)-based brain-computer interfaces (BCIs) for reducing calibration efforts.
However, backdoor attacks could be introduced through TL. In such attacks, an
attacker embeds a backdoor with a specific pattern into the machine learning
model. As a result, the model will misclassify a test sample with the backdoor
trigger into a prespecified class while still maintaining good performance on
benign samples. Accordingly, this study explores backdoor attacks in the TL of
EEG-based BCIs, where source-domain data are poisoned by a backdoor trigger and
then used in TL. We propose several active poisoning approaches to select
source-domain samples, which are most effective in embedding the backdoor
pattern, to improve the attack success rate and efficiency. Experiments on four
EEG datasets and three deep learning models demonstrate the effectiveness of
the approaches. To our knowledge, this is the first study about backdoor
attacks on TL models in EEG-based BCIs. It exposes a serious security risk in
BCIs, which should be immediately addressed.


http://arxiv.org/abs/2412.10265
Adversarial Robustness of Bottleneck Injected Deep Neural Networks for Task-Oriented Communication. (10%)
Alireza Furutanpey; Pantelis A. Frangoudis; Patrik Szabo; Schahram Dustdar
  This paper investigates the adversarial robustness of Deep Neural Networks
(DNNs) using Information Bottleneck (IB) objectives for task-oriented
communication systems. We empirically demonstrate that while IB-based
approaches provide baseline resilience against attacks targeting downstream
tasks, the reliance on generative models for task-oriented communication
introduces new vulnerabilities. Through extensive experiments on several
datasets, we analyze how bottleneck depth and task complexity influence
adversarial robustness. Our key findings show that Shallow Variational
Bottleneck Injection (SVBI) provides less adversarial robustness compared to
Deep Variational Information Bottleneck (DVIB) approaches, with the gap
widening for more complex tasks. Additionally, we reveal that IB-based
objectives exhibit stronger robustness against attacks focusing on salient
pixels with high intensity compared to those perturbing many pixels with lower
intensity. Lastly, we demonstrate that task-oriented communication systems that
rely on generative models to extract and recover salient information have an
increased attack surface. The results highlight important security
considerations for next-generation communication systems that leverage neural
networks for goal-oriented compression.


http://arxiv.org/abs/2412.10198
From Allies to Adversaries: Manipulating LLM Tool-Calling through Adversarial Injection. (1%)
Haowei Wang; Rupeng Zhang; Junjie Wang; Mingyang Li; Yuekai Huang; Dandan Wang; Qing Wang
  Tool-calling has changed Large Language Model (LLM) applications by
integrating external tools, significantly enhancing their functionality across
diverse tasks. However, this integration also introduces new security
vulnerabilities, particularly in the tool scheduling mechanisms of LLM, which
have not been extensively studied. To fill this gap, we present ToolCommander,
a novel framework designed to exploit vulnerabilities in LLM tool-calling
systems through adversarial tool injection. Our framework employs a
well-designed two-stage attack strategy. Firstly, it injects malicious tools to
collect user queries, then dynamically updates the injected tools based on the
stolen information to enhance subsequent attacks. These stages enable
ToolCommander to execute privacy theft, launch denial-of-service attacks, and
even manipulate business competition by triggering unscheduled tool-calling.
Notably, the ASR reaches 91.67% for privacy theft and hits 100% for
denial-of-service and unscheduled tool calling in certain cases. Our work
demonstrates that these vulnerabilities can lead to severe consequences beyond
simple misuse of tool-calling systems, underscoring the urgent need for robust
defensive strategies to secure LLM Tool-calling systems.


http://arxiv.org/abs/2412.10186
BiCert: A Bilinear Mixed Integer Programming Formulation for Precise Certified Bounds Against Data Poisoning Attacks. (1%)
Tobias Lorenz; Marta Kwiatkowska; Mario Fritz
  Data poisoning attacks pose one of the biggest threats to modern AI systems,
necessitating robust defenses. While extensive efforts have been made to
develop empirical defenses, attackers continue to evolve, creating
sophisticated methods to circumvent these measures. To address this, we must
move beyond empirical defenses and establish provable certification methods
that guarantee robustness. This paper introduces a novel certification
approach, BiCert, using Bilinear Mixed Integer Programming (BMIP) to compute
sound deterministic bounds that provide such provable robustness. Using BMIP,
we compute the reachable set of parameters that could result from training with
potentially manipulated data. A key element to make this computation feasible
is to relax the reachable parameter set to a convex set between training
iterations. At test time, this parameter set allows us to predict all possible
outcomes, guaranteeing robustness. BiCert is more precise than previous
methods, which rely solely on interval and polyhedral bounds. Crucially, our
approach overcomes the fundamental limitation of prior approaches where
parameter bounds could only grow, often uncontrollably. We show that BiCert's
tighter bounds eliminate a key source of divergence issues, resulting in more
stable training and higher certified accuracy.


http://arxiv.org/abs/2412.09692
Three-in-One: Robust Enhanced Universal Transferable Anti-Facial Retrieval in Online Social Networks. (99%)
Yunna Lv; Long Tang; Dengpan Ye; Caiyun Xie; Jiacheng Deng; Yiheng He
  Deep hash-based retrieval techniques are widely used in facial retrieval
systems to improve the efficiency of facial matching. However, it also carries
the danger of exposing private information. Deep hash models are easily
influenced by adversarial examples, which can be leveraged to protect private
images from malicious retrieval. The existing adversarial example methods
against deep hash models focus on universality and transferability, lacking the
research on its robustness in online social networks (OSNs), which leads to
their failure in anti-retrieval after post-processing. Therefore, we provide
the first in-depth discussion on robustness adversarial perturbation in
universal transferable anti-facial retrieval and propose Three-in-One
Adversarial Perturbation (TOAP). Specifically, we construct a local and global
Compression Generator (CG) to simulate complex post-processing scenarios, which
can be used to mitigate perturbation. Then, we propose robust optimization
objectives based on the discovery of the variation patterns of model's
distribution after post-processing, and generate adversarial examples using
these objectives and meta-learning. Finally, we iteratively optimize
perturbation by alternately generating adversarial examples and fine-tuning the
CG, balancing the performance of perturbation while enhancing CG's ability to
mitigate them. Numerous experiments demonstrate that, in addition to its
advantages in universality and transferability, TOAP significantly outperforms
current state-of-the-art methods in multiple robustness metrics. It further
improves universality and transferability by 5% to 28%, and achieves up to
about 33% significant improvement in several simulated post-processing
scenarios as well as mainstream OSNs, demonstrating that TOAP can effectively
protect private images from malicious retrieval in real-world scenarios.


http://arxiv.org/abs/2412.09195
On the Generation and Removal of Speaker Adversarial Perturbation for Voice-Privacy Protection. (95%)
Chenyang Guo; Liping Chen; Zhuhai Li; Kong Aik Lee; Zhen-Hua Ling; Wu Guo
  Neural networks are commonly known to be vulnerable to adversarial attacks
mounted through subtle perturbation on the input data. Recent development in
voice-privacy protection has shown the positive use cases of the same technique
to conceal speaker's voice attribute with additive perturbation signal
generated by an adversarial network. This paper examines the reversibility
property where an entity generating the adversarial perturbations is authorized
to remove them and restore original speech (e.g., the speaker him/herself). A
similar technique could also be used by an investigator to deanonymize a
voice-protected speech to restore criminals' identities in security and
forensic analysis. In this setting, the perturbation generative module is
assumed to be known in the removal process. To this end, a joint training of
perturbation generation and removal modules is proposed. Experimental results
on the LibriSpeech dataset demonstrated that the subtle perturbations added to
the original speech can be predicted from the anonymized speech while achieving
the goal of privacy protection. By removing these perturbations from the
anonymized sample, the original speech can be restored. Audio samples can be
found in \url{https://voiceprivacy.github.io/Perturbation-Generation-Removal/}.


http://arxiv.org/abs/2412.09844
Real-time Identity Defenses against Malicious Personalization of Diffusion Models. (95%)
Hanzhong Guo; Shen Nie; Chao Du; Tianyu Pang; Hao Sun; Chongxuan Li
  Personalized diffusion models, capable of synthesizing highly realistic
images based on a few reference portraits, pose substantial social, ethical,
and legal risks by enabling identity replication. Existing defense mechanisms
rely on computationally intensive adversarial perturbations tailored to
individual images, rendering them impractical for real-world deployment. This
study introduces Real-time Identity Defender (RID), a neural network designed
to generate adversarial perturbations through a single forward pass, bypassing
the need for image-specific optimization. RID achieves unprecedented
efficiency, with defense times as low as 0.12 seconds on a single GPU (4,400
times faster than leading methods) and 1.1 seconds per image on a standard
Intel i9 CPU, making it suitable for edge devices such as smartphones. Despite
its efficiency, RID matches state-of-the-art performance across visual and
quantitative benchmarks, effectively mitigating identity replication risks. Our
analysis reveals that RID's perturbations mimic the efficacy of traditional
defenses while exhibiting properties distinct from natural noise, such as
Gaussian perturbations. To enhance robustness, we extend RID into an ensemble
framework that integrates multiple pre-trained text-to-image diffusion models,
ensuring resilience against black-box attacks and post-processing techniques,
including JPEG compression and diffusion-based purification.


http://arxiv.org/abs/2412.08969
Deep Learning Model Security: Threats and Defenses. (92%)
Tianyang Wang; Ziqian Bi; Yichao Zhang; Ming Liu; Weiche Hsieh; Pohsun Feng; Lawrence K. Q. Yan; Yizhu Wen; Benji Peng; Junyu Liu; Keyu Chen; Sen Zhang; Ming Li; Chuanqi Jiang; Xinyuan Song; Junjie Yang; Bowen Jing; Jintao Ren; Junhao Song; Hong-Ming Tseng; Silin Chen; Yunze Wang; Chia Xin Liang; Jiawei Xu; Xuanhe Pan; Jinlang Wang; Qian Niu
  Deep learning has transformed AI applications but faces critical security
challenges, including adversarial attacks, data poisoning, model theft, and
privacy leakage. This survey examines these vulnerabilities, detailing their
mechanisms and impact on model integrity and confidentiality. Practical
implementations, including adversarial examples, label flipping, and backdoor
attacks, are explored alongside defenses such as adversarial training,
differential privacy, and federated learning, highlighting their strengths and
limitations.
  Advanced methods like contrastive and self-supervised learning are presented
for enhancing robustness. The survey concludes with future directions,
emphasizing automated defenses, zero-trust architectures, and the security
challenges of large AI models. A balanced approach to performance and security
is essential for developing reliable deep learning systems.


http://arxiv.org/abs/2412.09450
A Semi Black-Box Adversarial Bit-Flip Attack with Limited DNN Model Information. (69%)
Behnam Ghavami; Mani Sadati; Mohammad Shahidzadeh; Lesley Shannon; Steve Wilton
  Despite the rising prevalence of deep neural networks (DNNs) in
cyber-physical systems, their vulnerability to adversarial bit-flip attacks
(BFAs) is a noteworthy concern. This paper proposes B3FA, a semi-black-box
BFA-based parameter attack on DNNs, assuming the adversary has limited
knowledge about the model. We consider practical scenarios often feature a more
restricted threat model for real-world systems, contrasting with the typical
BFA models that presuppose the adversary's full access to a network's inputs
and parameters. The introduced bit-flip approach utilizes a magnitude-based
ranking method and a statistical re-construction technique to identify the
vulnerable bits. We demonstrate the effectiveness of B3FA on several DNN models
in a semi-black-box setting. For example, B3FA could drop the accuracy of a
MobileNetV2 from 69.84% to 9% with only 20 bit-flips in a real-world setting.


http://arxiv.org/abs/2412.09150
Evaluating Adversarial Attacks on Traffic Sign Classifiers beyond Standard Baselines. (45%)
Svetlana Pavlitska; Leopold Müller; J. Marius Zöllner
  Adversarial attacks on traffic sign classification models were among the
first successfully tried in the real world. Since then, the research in this
area has been mainly restricted to repeating baseline models, such as LISA-CNN
or GTSRB-CNN, and similar experiment settings, including white and black
patches on traffic signs. In this work, we decouple model architectures from
the datasets and evaluate on further generic models to make a fair comparison.
Furthermore, we compare two attack settings, inconspicuous and visible, which
are usually regarded without direct comparison. Our results show that standard
baselines like LISA-CNN or GTSRB-CNN are significantly more susceptible than
the generic ones. We, therefore, suggest evaluating new attacks on a broader
spectrum of baselines in the future. Our code is available at
\url{https://github.com/KASTEL-MobilityLab/attacks-on-traffic-sign-recognition/}.


http://arxiv.org/abs/2412.09073
SVasP: Self-Versatility Adversarial Style Perturbation for Cross-Domain Few-Shot Learning. (3%)
Wenqian Li; Pengfei Fang; Hui Xue
  Cross-Domain Few-Shot Learning (CD-FSL) aims to transfer knowledge from seen
source domains to unseen target domains, which is crucial for evaluating the
generalization and robustness of models. Recent studies focus on utilizing
visual styles to bridge the domain gap between different domains. However, the
serious dilemma of gradient instability and local optimization problem occurs
in those style-based CD-FSL methods. This paper addresses these issues and
proposes a novel crop-global style perturbation method, called
\underline{\textbf{S}}elf-\underline{\textbf{V}}ersatility
\underline{\textbf{A}}dversarial \underline{\textbf{S}}tyle
\underline{\textbf{P}}erturbation (\textbf{SVasP}), which enhances the gradient
stability and escapes from poor sharp minima jointly. Specifically, SVasP
simulates more diverse potential target domain adversarial styles via
diversifying input patterns and aggregating localized crop style gradients, to
serve as global style perturbation stabilizers within one image, a concept we
refer to as self-versatility. Then a novel objective function is proposed to
maximize visual discrepancy while maintaining semantic consistency between
global, crop, and adversarial features. Having the stabilized global style
perturbation in the training phase, one can obtain a flattened minima in the
loss landscape, boosting the transferability of the model to the target
domains. Extensive experiments on multiple benchmark datasets demonstrate that
our method significantly outperforms existing state-of-the-art methods. Our
codes are available at https://github.com/liwenqianSEU/SVasP.


http://arxiv.org/abs/2412.09565
Obfuscated Activations Bypass LLM Latent-Space Defenses. (2%)
Luke Bailey; Alex Serrano; Abhay Sheshadri; Mikhail Seleznyov; Jordan Taylor; Erik Jenner; Jacob Hilton; Stephen Casper; Carlos Guestrin; Scott Emmons
  Recent latent-space monitoring techniques have shown promise as defenses
against LLM attacks. These defenses act as scanners that seek to detect harmful
activations before they lead to undesirable actions. This prompts the question:
Can models execute harmful behavior via inconspicuous latent states? Here, we
study such obfuscated activations. We show that state-of-the-art latent-space
defenses -- including sparse autoencoders, representation probing, and latent
OOD detection -- are all vulnerable to obfuscated activations. For example,
against probes trained to classify harmfulness, our attacks can often reduce
recall from 100% to 0% while retaining a 90% jailbreaking rate. However,
obfuscation has limits: we find that on a complex task (writing SQL code),
obfuscation reduces model performance. Together, our results demonstrate that
neural activations are highly malleable: we can reshape activation patterns in
a variety of ways, often while preserving a network's behavior. This poses a
fundamental challenge to latent-space defenses.


http://arxiv.org/abs/2412.09269
Towards Understanding the Robustness of LLM-based Evaluations under Perturbations. (1%)
Manav Chaudhary; Harshit Gupta; Savita Bhat; Vasudeva Varma
  Traditional evaluation metrics like BLEU and ROUGE fall short when capturing
the nuanced qualities of generated text, particularly when there is no single
ground truth. In this paper, we explore the potential of Large Language Models
(LLMs), specifically Google Gemini 1, to serve as automatic evaluators for
non-standardized metrics in summarization and dialog-based tasks. We conduct
experiments across multiple prompting strategies to examine how LLMs fare as
quality evaluators when compared with human judgments on the SummEval and USR
datasets, asking the model to generate both a score as well as a justification
for the score. Furthermore, we explore the robustness of the LLM evaluator by
using perturbed inputs. Our findings suggest that while LLMs show promise,
their alignment with human evaluators is limited, they are not robust against
perturbations and significant improvements are required for their standalone
use as reliable evaluators for subjective metrics.


http://arxiv.org/abs/2412.09765
L-WISE: Boosting Human Image Category Learning Through Model-Based Image Selection And Enhancement. (1%)
Morgan B. Talbot; Gabriel Kreiman; James J. DiCarlo; Guy Gaziv
  The currently leading artificial neural network (ANN) models of the visual
ventral stream -- which are derived from a combination of performance
optimization and robustification methods -- have demonstrated a remarkable
degree of behavioral alignment with humans on visual categorization tasks.
Extending upon previous work, we show that not only can these models guide
image perturbations that change the induced human category percepts, but they
also can enhance human ability to accurately report the original ground truth.
Furthermore, we find that the same models can also be used out-of-the-box to
predict the proportion of correct human responses to individual images,
providing a simple, human-aligned estimator of the relative difficulty of each
image. Motivated by these observations, we propose to augment visual learning
in humans in a way that improves human categorization accuracy at test time.
Our learning augmentation approach consists of (i) selecting images based on
their model-estimated recognition difficulty, and (ii) using image
perturbations that aid recognition for novice learners. We find that combining
these model-based strategies gives rise to test-time categorization accuracy
gains of 33-72% relative to control subjects without these interventions,
despite using the same number of training feedback trials. Surprisingly, beyond
the accuracy gain, the training time for the augmented learning group was also
shorter by 20-23%. We demonstrate the efficacy of our approach in a
fine-grained categorization task with natural images, as well as tasks in two
clinically relevant image domains -- histology and dermoscopy -- where visual
learning is notoriously challenging. To the best of our knowledge, this is the
first application of ANNs to increase visual learning performance in humans by
enhancing category-specific features.


http://arxiv.org/abs/2412.08394
Adversarial Purification by Consistency-aware Latent Space Optimization on Data Manifolds. (99%)
Shuhai Zhang; Jiahao Yang; Hui Luo; Jie Chen; Li Wang; Feng Liu; Bo Han; Mingkui Tan
  Deep neural networks (DNNs) are vulnerable to adversarial samples crafted by
adding imperceptible perturbations to clean data, potentially leading to
incorrect and dangerous predictions. Adversarial purification has been an
effective means to improve DNNs robustness by removing these perturbations
before feeding the data into the model. However, it faces significant
challenges in preserving key structural and semantic information of data, as
the imperceptible nature of adversarial perturbations makes it hard to avoid
over-correcting, which can destroy important information and degrade model
performance. In this paper, we break away from traditional adversarial
purification methods by focusing on the clean data manifold. To this end, we
reveal that samples generated by a well-trained generative model are close to
clean ones but far from adversarial ones. Leveraging this insight, we propose
Consistency Model-based Adversarial Purification (CMAP), which optimizes
vectors within the latent space of a pre-trained consistency model to generate
samples for restoring clean data. Specifically, 1) we propose a
\textit{Perceptual consistency restoration} mechanism by minimizing the
discrepancy between generated samples and input samples in both pixel and
perceptual spaces. 2) To maintain the optimized latent vectors within the valid
data manifold, we introduce a \textit{Latent distribution consistency
constraint} strategy to align generated samples with the clean data
distribution. 3) We also apply a \textit{Latent vector consistency prediction}
scheme via an ensemble approach to enhance prediction reliability. CMAP
fundamentally addresses adversarial perturbations at their source, providing a
robust purification. Extensive experiments on CIFAR-10 and ImageNet-100 show
that our CMAP significantly enhances robustness against strong adversarial
attacks while preserving high natural accuracy.


http://arxiv.org/abs/2412.08108
Doubly-Universal Adversarial Perturbations: Deceiving Vision-Language Models Across Both Images and Text with a Single Perturbation. (98%)
Hee-Seon Kim; Minbeom Kim; Changick Kim
  Large Vision-Language Models (VLMs) have demonstrated remarkable performance
across multimodal tasks by integrating vision encoders with large language
models (LLMs). However, these models remain vulnerable to adversarial attacks.
Among such attacks, Universal Adversarial Perturbations (UAPs) are especially
powerful, as a single optimized perturbation can mislead the model across
various input images. In this work, we introduce a novel UAP specifically
designed for VLMs: the Doubly-Universal Adversarial Perturbation (Doubly-UAP),
capable of universally deceiving VLMs across both image and text inputs. To
successfully disrupt the vision encoder's fundamental process, we analyze the
core components of the attention mechanism. After identifying value vectors in
the middle-to-late layers as the most vulnerable, we optimize Doubly-UAP in a
label-free manner with a frozen model. Despite being developed as a black-box
to the LLM, Doubly-UAP achieves high attack success rates on VLMs, consistently
outperforming baseline methods across vision-language tasks. Extensive ablation
studies and analyses further demonstrate the robustness of Doubly-UAP and
provide insights into how it influences internal attention mechanisms.


http://arxiv.org/abs/2412.08555
Grimm: A Plug-and-Play Perturbation Rectifier for Graph Neural Networks Defending against Poisoning Attacks. (93%)
Ao Liu; Wenshan Li; Beibei Li; Wengang Ma; Tao Li; Pan Zhou
  Recent studies have revealed the vulnerability of graph neural networks
(GNNs) to adversarial poisoning attacks on node classification tasks. Current
defensive methods require substituting the original GNNs with defense models,
regardless of the original's type. This approach, while targeting adversarial
robustness, compromises the enhancements developed in prior research to boost
GNNs' practical performance. Here we introduce Grimm, the first plug-and-play
defense model. With just a minimal interface requirement for extracting
features from any layer of the protected GNNs, Grimm is thus enabled to
seamlessly rectify perturbations. Specifically, we utilize the feature
trajectories (FTs) generated by GNNs, as they evolve through epochs, to reflect
the training status of the networks. We then theoretically prove that the FTs
of victim nodes will inevitably exhibit discriminable anomalies. Consequently,
inspired by the natural parallelism between the biological nervous and immune
systems, we construct Grimm, a comprehensive artificial immune system for GNNs.
Grimm not only detects abnormal FTs and rectifies adversarial edges during
training but also operates efficiently in parallel, thereby mirroring the
concurrent functionalities of its biological counterparts. We experimentally
confirm that Grimm offers four empirically validated advantages: 1)
Harmlessness, as it does not actively interfere with GNN training; 2)
Parallelism, ensuring monitoring, detection, and rectification functions
operate independently of the GNN training process; 3) Generalizability,
demonstrating compatibility with mainstream GNNs such as GCN, GAT, and
GraphSAGE; and 4) Transferability, as the detectors for abnormal FTs can be
efficiently transferred across different systems for one-step rectification.


http://arxiv.org/abs/2412.08615
Exploiting the Index Gradients for Optimization-Based Jailbreaking on Large Language Models. (83%)
Jiahui Li; Yongchang Hao; Haoyu Xu; Xing Wang; Yu Hong
  Despite the advancements in training Large Language Models (LLMs) with
alignment techniques to enhance the safety of generated content, these models
remain susceptible to jailbreak, an adversarial attack method that exposes
security vulnerabilities in LLMs. Notably, the Greedy Coordinate Gradient (GCG)
method has demonstrated the ability to automatically generate adversarial
suffixes that jailbreak state-of-the-art LLMs. However, the optimization
process involved in GCG is highly time-consuming, rendering the jailbreaking
pipeline inefficient. In this paper, we investigate the process of GCG and
identify an issue of Indirect Effect, the key bottleneck of the GCG
optimization. To this end, we propose the Model Attack Gradient Index GCG
(MAGIC), that addresses the Indirect Effect by exploiting the gradient
information of the suffix tokens, thereby accelerating the procedure by having
less computation and fewer iterations. Our experiments on AdvBench show that
MAGIC achieves up to a 1.5x speedup, while maintaining Attack Success Rates
(ASR) on par or even higher than other baselines. Our MAGIC achieved an ASR of
74% on the Llama-2 and an ASR of 54% when conducting transfer attacks on
GPT-3.5. Code is available at https://github.com/jiah-li/magic.


http://arxiv.org/abs/2412.08608
AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Models. (82%)
Mintong Kang; Chejian Xu; Bo Li
  Recent advancements in large audio-language models (LALMs) have enabled
speech-based user interactions, significantly enhancing user experience and
accelerating the deployment of LALMs in real-world applications. However,
ensuring the safety of LALMs is crucial to prevent risky outputs that may raise
societal concerns or violate AI regulations. Despite the importance of this
issue, research on jailbreaking LALMs remains limited due to their recent
emergence and the additional technical challenges they present compared to
attacks on DNN-based audio models. Specifically, the audio encoders in LALMs,
which involve discretization operations, often lead to gradient shattering,
hindering the effectiveness of attacks relying on gradient-based optimizations.
The behavioral variability of LALMs further complicates the identification of
effective (adversarial) optimization targets. Moreover, enforcing stealthiness
constraints on adversarial audio waveforms introduces a reduced, non-convex
feasible solution space, further intensifying the challenges of the
optimization process. To overcome these challenges, we develop AdvWave, the
first jailbreak framework against LALMs. We propose a dual-phase optimization
method that addresses gradient shattering, enabling effective end-to-end
gradient-based optimization. Additionally, we develop an adaptive adversarial
target search algorithm that dynamically adjusts the adversarial optimization
target based on the response patterns of LALMs for specific queries. To ensure
that adversarial audio remains perceptually natural to human listeners, we
design a classifier-guided optimization approach that generates adversarial
noise resembling common urban sounds. Extensive evaluations on multiple
advanced LALMs demonstrate that AdvWave outperforms baseline methods, achieving
a 40% higher average jailbreak attack success rate.


http://arxiv.org/abs/2412.08755
Proactive Adversarial Defense: Harnessing Prompt Tuning in Vision-Language Models to Detect Unseen Backdoored Images. (45%)
Kyle Stein; Andrew Arash Mahyari; Guillermo Francia; Eman El-Sheikh
  Backdoor attacks pose a critical threat by embedding hidden triggers into
inputs, causing models to misclassify them into target labels. While extensive
research has focused on mitigating these attacks in object recognition models
through weight fine-tuning, much less attention has been given to detecting
backdoored samples directly. Given the vast datasets used in training, manual
inspection for backdoor triggers is impractical, and even state-of-the-art
defense mechanisms fail to fully neutralize their impact. To address this gap,
we introduce a groundbreaking method to detect unseen backdoored images during
both training and inference. Leveraging the transformative success of prompt
tuning in Vision Language Models (VLMs), our approach trains learnable text
prompts to differentiate clean images from those with hidden backdoor triggers.
Experiments demonstrate the exceptional efficacy of this method, achieving an
impressive average accuracy of 86% across two renowned datasets for detecting
unseen backdoor triggers, establishing a new standard in backdoor defense.


http://arxiv.org/abs/2412.08366
Backdoor attacks on DNN and GBDT -- A Case Study from the insurance domain. (16%)
Robin Debeka, Koblenz, Germany Kühlem; Daniel Debeka, Koblenz, Germany Otten; Daniel Debeka, Koblenz, Germany Ludwig; Anselm Debeka, Koblenz, Germany Department of Maths and Technology, Koblenz University of Applied Sciences, Remagen, Germany Hudde; Alexander Computer Science, University of Koblenz, Koblenz, Germany Rosenbaum; Andreas Computer Science, University of Koblenz, Koblenz, Germany Mauthe
  Machine learning (ML) will likely play a large role in many processes in the
future, also for insurance companies. However, ML models are at risk of being
attacked and manipulated. In this work, the robustness of Gradient Boosted
Decision Tree (GBDT) models and Deep Neural Networks (DNN) within an insurance
context will be evaluated. Therefore, two GBDT models and two DNNs are trained
on two different tabular datasets from an insurance context. Past research in
this domain mainly used homogenous data and there are comparably few insights
regarding heterogenous tabular data. The ML tasks performed on the datasets are
claim prediction (regression) and fraud detection (binary classification). For
the backdoor attacks different samples containing a specific pattern were
crafted and added to the training data. It is shown, that this type of attack
can be highly successful, even with a few added samples. The backdoor attacks
worked well on the models trained on one dataset but poorly on the models
trained on the other. In real-world scenarios the attacker will have to face
several obstacles but as attacks can work with very few added samples this risk
should be evaluated.


http://arxiv.org/abs/2412.08156
Antelope: Potent and Concealed Jailbreak Attack Strategy. (10%)
Xin Zhao; Xiaojun Chen; Haoyu Gao
  Due to the remarkable generative potential of diffusion-based models,
numerous researches have investigated jailbreak attacks targeting these
frameworks. A particularly concerning threat within image models is the
generation of Not-Safe-for-Work (NSFW) content. Despite the implementation of
security filters, numerous efforts continue to explore ways to circumvent these
safeguards. Current attack methodologies primarily encompass adversarial prompt
engineering or concept obfuscation, yet they frequently suffer from slow search
efficiency, conspicuous attack characteristics and poor alignment with targets.
To overcome these challenges, we propose Antelope, a more robust and covert
jailbreak attack strategy designed to expose security vulnerabilities inherent
in generative models. Specifically, Antelope leverages the confusion of
sensitive concepts with similar ones, facilitates searches in the semantically
adjacent space of these related concepts and aligns them with the target
imagery, thereby generating sensitive images that are consistent with the
target and capable of evading detection. Besides, we successfully exploit the
transferability of model-based attacks to penetrate online black-box services.
Experimental evaluations demonstrate that Antelope outperforms existing
baselines across multiple defensive mechanisms, underscoring its efficacy and
versatility.


http://arxiv.org/abs/2412.08201
Model-Editing-Based Jailbreak against Safety-aligned Large Language Models. (1%)
Yuxi Li; Zhibo Zhang; Kailong Wang; Ling Shi; Haoyu Wang
  Large Language Models (LLMs) have transformed numerous fields by enabling
advanced natural language interactions but remain susceptible to critical
vulnerabilities, particularly jailbreak attacks. Current jailbreak techniques,
while effective, often depend on input modifications, making them detectable
and limiting their stealth and scalability. This paper presents Targeted Model
Editing (TME), a novel white-box approach that bypasses safety filters by
minimally altering internal model structures while preserving the model's
intended functionalities. TME identifies and removes safety-critical
transformations (SCTs) embedded in model matrices, enabling malicious queries
to bypass restrictions without input modifications. By analyzing distinct
activation patterns between safe and unsafe queries, TME isolates and
approximates SCTs through an optimization process. Implemented in the D-LLM
framework, our method achieves an average Attack Success Rate (ASR) of 84.86%
on four mainstream open-source LLMs, maintaining high performance. Unlike
existing methods, D-LLM eliminates the need for specific triggers or harmful
response collections, offering a stealthier and more effective jailbreak
strategy. This work reveals a covert and robust threat vector in LLM security
and emphasizes the need for stronger safeguards in model safety alignment.


http://arxiv.org/abs/2412.07468
AHSG: Adversarial Attacks on High-level Semantics in Graph Neural Networks. (99%)
Kai Yuan; Xiaobing Pei; Haoran Yang
  Graph Neural Networks (GNNs) have garnered significant interest among
researchers due to their impressive performance in graph learning tasks.
However, like other deep neural networks, GNNs are also vulnerable to
adversarial attacks. In existing adversarial attack methods for GNNs, the
metric between the attacked graph and the original graph is usually the attack
budget or a measure of global graph properties. However, we have found that it
is possible to generate attack graphs that disrupt the primary semantics even
within these constraints. To address this problem, we propose a Adversarial
Attacks on High-level Semantics in Graph Neural Networks (AHSG), which is a
graph structure attack model that ensures the retention of primary semantics.
The latent representations of each node can extract rich semantic information
by applying convolutional operations on graph data. These representations
contain both task-relevant primary semantic information and task-irrelevant
secondary semantic information. The latent representations of same-class nodes
with the same primary semantics can fulfill the objective of modifying
secondary semantics while preserving the primary semantics. Finally, the latent
representations with attack effects is mapped to an attack graph using
Projected Gradient Descent (PGD) algorithm. By attacking graph deep learning
models with some advanced defense strategies, we validate that AHSG has
superior attack effectiveness compared to other attack methods. Additionally,
we employ Contextual Stochastic Block Models (CSBMs) as a proxy for the primary
semantics to detect the attacked graph, confirming that AHSG almost does not
disrupt the original primary semantics of the graph.


http://arxiv.org/abs/2412.07326
Addressing Key Challenges of Adversarial Attacks and Defenses in the Tabular Domain: A Methodological Framework for Coherence and Consistency. (99%)
Yael Itzhakev; Amit Giloni; Yuval Elovici; Asaf Shabtai
  Machine learning models trained on tabular data are vulnerable to adversarial
attacks, even in realistic scenarios where attackers have access only to the
model's outputs. Researchers evaluate such attacks by considering metrics like
success rate, perturbation magnitude, and query count. However, unlike other
data domains, the tabular domain contains complex interdependencies among
features, presenting a unique aspect that should be evaluated: the need for the
attack to generate coherent samples and ensure feature consistency for
indistinguishability. Currently, there is no established methodology for
evaluating adversarial samples based on these criteria. In this paper, we
address this gap by proposing new evaluation criteria tailored for tabular
attacks' quality; we defined anomaly-based framework to assess the
distinguishability of adversarial samples and utilize the SHAP explainability
technique to identify inconsistencies in the model's decision-making process
caused by adversarial samples. These criteria could form the basis for
potential detection methods and be integrated into established evaluation
metrics for assessing attack's quality Additionally, we introduce a novel
technique for perturbing dependent features while maintaining coherence and
feature consistency within the sample. We compare different attacks'
strategies, examining black-box query-based attacks and transferability-based
gradient attacks across four target models. Our experiments, conducted on
benchmark tabular datasets, reveal significant differences between the examined
attacks' strategies in terms of the attacker's risk and effort and the attacks'
quality. The findings provide valuable insights on the strengths, limitations,
and trade-offs of various adversarial attacks in the tabular domain, laying a
foundation for future research on attacks and defense development.


http://arxiv.org/abs/2412.07274
A Generative Victim Model for Segmentation. (99%)
Aixuan Li; Jing Zhang; Jiawei Shi; Yiran Zhong; Yuchao Dai
  We find that the well-trained victim models (VMs), against which the attacks
are generated, serve as fundamental prerequisites for adversarial attacks, i.e.
a segmentation VM is needed to generate attacks for segmentation. In this
context, the victim model is assumed to be robust to achieve effective
adversarial perturbation generation. Instead of focusing on improving the
robustness of the task-specific victim models, we shift our attention to image
generation. From an image generation perspective, we derive a novel VM for
segmentation, aiming to generate adversarial perturbations for segmentation
tasks without requiring models explicitly designed for image segmentation. Our
approach to adversarial attack generation diverges from conventional white-box
or black-box attacks, offering a fresh outlook on adversarial attack
strategies. Experiments show that our attack method is able to generate
effective adversarial attacks with good transferability.


http://arxiv.org/abs/2412.07277
Backdoor Attacks against No-Reference Image Quality Assessment Models via a Scalable Trigger. (99%)
Yi Yu; Song Xia; Xun Lin; Wenhan Yang; Shijian Lu; Yap-peng Tan; Alex Kot
  No-Reference Image Quality Assessment (NR-IQA), responsible for assessing the
quality of a single input image without using any reference, plays a critical
role in evaluating and optimizing computer vision systems, e.g., low-light
enhancement. Recent research indicates that NR-IQA models are susceptible to
adversarial attacks, which can significantly alter predicted scores with
visually imperceptible perturbations. Despite revealing vulnerabilities, these
attack methods have limitations, including high computational demands,
untargeted manipulation, limited practical utility in white-box scenarios, and
reduced effectiveness in black-box scenarios. To address these challenges, we
shift our focus to another significant threat and present a novel
poisoning-based backdoor attack against NR-IQA (BAIQA), allowing the attacker
to manipulate the IQA model's output to any desired target value by simply
adjusting a scaling coefficient $\alpha$ for the trigger. We propose to inject
the trigger in the discrete cosine transform (DCT) domain to improve the local
invariance of the trigger for countering trigger diminishment in NR-IQA models
due to widely adopted data augmentations. Furthermore, the universal
adversarial perturbations (UAP) in the DCT space are designed as the trigger,
to increase IQA model susceptibility to manipulation and improve attack
effectiveness. In addition to the heuristic method for poison-label BAIQA
(P-BAIQA), we explore the design of clean-label BAIQA (C-BAIQA), focusing on
$\alpha$ sampling and image data refinement, driven by theoretical insights we
reveal. Extensive experiments on diverse datasets and various NR-IQA models
demonstrate the effectiveness of our attacks. Code can be found at
https://github.com/yuyi-sd/BAIQA.


http://arxiv.org/abs/2412.07575
Defending Against Neural Network Model Inversion Attacks via Data Poisoning. (98%)
Shuai Zhou; Dayong Ye; Tianqing Zhu; Wanlei Zhou
  Model inversion attacks pose a significant privacy threat to machine learning
models by reconstructing sensitive data from their outputs. While various
defenses have been proposed to counteract these attacks, they often come at the
cost of the classifier's utility, thus creating a challenging trade-off between
privacy protection and model utility. Moreover, most existing defenses require
retraining the classifier for enhanced robustness, which is impractical for
large-scale, well-established models. This paper introduces a novel defense
mechanism to better balance privacy and utility, particularly against
adversaries who employ a machine learning model (i.e., inversion model) to
reconstruct private data. Drawing inspiration from data poisoning attacks,
which can compromise the performance of machine learning models, we propose a
strategy that leverages data poisoning to contaminate the training data of
inversion models, thereby preventing model inversion attacks.
  Two defense methods are presented. The first, termed label-preserving
poisoning attacks for all output vectors (LPA), involves subtle perturbations
to all output vectors while preserving their labels. Our findings demonstrate
that these minor perturbations, introduced through a data poisoning approach,
significantly increase the difficulty of data reconstruction without
compromising the utility of the classifier. Subsequently, we introduce a second
method, label-flipping poisoning for partial output vectors (LFP), which
selectively perturbs a small subset of output vectors and alters their labels
during the process. Empirical results indicate that LPA is notably effective,
outperforming the current state-of-the-art defenses. Our data poisoning-based
defense provides a new retraining-free defense paradigm that preserves the
victim classifier's utility.


http://arxiv.org/abs/2412.08099
Adversarial Vulnerabilities in Large Language Models for Time Series Forecasting. (98%)
Fuqiang Liu; Sicong Jiang; Luis Miranda-Moreno; Seongjin Choi; Lijun Sun
  Large Language Models (LLMs) have recently demonstrated significant potential
in the field of time series forecasting, offering impressive capabilities in
handling complex temporal data. However, their robustness and reliability in
real-world applications remain under-explored, particularly concerning their
susceptibility to adversarial attacks. In this paper, we introduce a targeted
adversarial attack framework for LLM-based time series forecasting. By
employing both gradient-free and black-box optimization methods, we generate
minimal yet highly effective perturbations that significantly degrade the
forecasting accuracy across multiple datasets and LLM architectures. Our
experiments, which include models like TimeGPT and LLM-Time with GPT-3.5,
GPT-4, LLaMa, and Mistral, show that adversarial attacks lead to much more
severe performance degradation than random noise, and demonstrate the broad
effectiveness of our attacks across different LLMs. The results underscore the
critical vulnerabilities of LLMs in time series forecasting, highlighting the
need for robust defense mechanisms to ensure their reliable deployment in
practical applications.


http://arxiv.org/abs/2412.08053
DynamicPAE: Generating Scene-Aware Physical Adversarial Examples in Real-Time. (92%)
Jin Hu; Xianglong Liu; Jiakai Wang; Junkai Zhang; Xianqi Yang; Haotong Qin; Yuqing Ma; Ke Xu
  Physical adversarial examples (PAEs) are regarded as "whistle-blowers" of
real-world risks in deep-learning applications. However, current PAE generation
studies show limited adaptive attacking ability to diverse and varying scenes.
The key challenges in generating dynamic PAEs are exploring their patterns
under noisy gradient feedback and adapting the attack to agnostic scenario
natures. To address the problems, we present DynamicPAE, the first generative
framework that enables scene-aware real-time physical attacks beyond static
attacks. Specifically, to train the dynamic PAE generator under noisy gradient
feedback, we introduce the residual-driven sample trajectory guidance
technique, which redefines the training task to break the limited feedback
information restriction that leads to the degeneracy problem. Intuitively, it
allows the gradient feedback to be passed to the generator through a low-noise
auxiliary task, thereby guiding the optimization away from degenerate solutions
and facilitating a more comprehensive and stable exploration of feasible PAEs.
To adapt the generator to agnostic scenario natures, we introduce the
context-aligned scene expectation simulation process, consisting of the
conditional-uncertainty-aligned data module and the skewness-aligned objective
re-weighting module. The former enhances robustness in the context of
incomplete observation by employing a conditional probabilistic model for
domain randomization, while the latter facilitates consistent stealth control
across different attack targets by automatically reweighting losses based on
the skewness indicator. Extensive digital and physical evaluations demonstrate
the superior attack performance of DynamicPAE, attaining a 1.95 $\times$ boost
(65.55% average AP drop under attack) on representative object detectors (e.g.,
Yolo-v8) over state-of-the-art static PAE generating methods.


http://arxiv.org/abs/2412.07559
Adaptive Epsilon Adversarial Training for Robust Gravitational Wave Parameter Estimation Using Normalizing Flows. (86%)
Yiqian Yang; Xihua Zhu; Fan Zhang
  Adversarial training with Normalizing Flow (NF) models is an emerging
research area aimed at improving model robustness through adversarial samples.
In this study, we focus on applying adversarial training to NF models for
gravitational wave parameter estimation. We propose an adaptive epsilon method
for Fast Gradient Sign Method (FGSM) adversarial training, which dynamically
adjusts perturbation strengths based on gradient magnitudes using logarithmic
scaling. Our hybrid architecture, combining ResNet and Inverse Autoregressive
Flow, reduces the Negative Log Likelihood (NLL) loss by 47\% under FGSM attacks
compared to the baseline model, while maintaining an NLL of 4.2 on clean data
(only 5\% higher than the baseline). For perturbation strengths between 0.01
and 0.1, our model achieves an average NLL of 5.8, outperforming both
fixed-epsilon (NLL: 6.7) and progressive-epsilon (NLL: 7.2) methods. Under
stronger Projected Gradient Descent attacks with perturbation strength of 0.05,
our model maintains an NLL of 6.4, demonstrating superior robustness while
avoiding catastrophic overfitting.


http://arxiv.org/abs/2412.08098
What You See Is Not Always What You Get: An Empirical Study of Code Comprehension by Large Language Models. (83%)
Bangshuo Zhu; Jiawen Wen; Huaming Chen
  Recent studies have demonstrated outstanding capabilities of large language
models (LLMs) in software engineering domain, covering numerous tasks such as
code generation and comprehension. While the benefit of LLMs for coding task is
well noted, it is perceived that LLMs are vulnerable to adversarial attacks. In
this paper, we study the specific LLM vulnerability to imperceptible character
attacks, a type of prompt-injection attack that uses special characters to
befuddle an LLM whilst keeping the attack hidden to human eyes. We devise four
categories of attacks and investigate their effects on the performance outcomes
of tasks relating to code analysis and code comprehension. Two generations of
ChatGPT are included to evaluate the impact of advancements made to
contemporary models. Our experimental design consisted of comparing perturbed
and unperturbed code snippets and evaluating two performance outcomes, which
are model confidence using log probabilities of response, and correctness of
response. We conclude that earlier version of ChatGPT exhibits a strong
negative linear correlation between the amount of perturbation and the
performance outcomes, while the recent ChatGPT presents a strong negative
correlation between the presence of perturbation and performance outcomes, but
no valid correlational relationship between perturbation budget and performance
outcomes. We anticipate this work contributes to an in-depth understanding of
leveraging LLMs for coding tasks. It is suggested future research should delve
into how to create LLMs that can return a correct response even if the prompt
exhibits perturbations.


http://arxiv.org/abs/2412.08014
MAGIC: Mastering Physical Adversarial Generation in Context through Collaborative LLM Agents. (82%)
Yun Xing; Nhat Chung; Jie Zhang; Yue Cao; Ivor Tsang; Yang Liu; Lei Ma; Qing Guo
  Physical adversarial attacks in driving scenarios can expose critical
vulnerabilities in visual perception models. However, developing such attacks
remains challenging due to diverse real-world backgrounds and the requirement
for maintaining visual naturality. Building upon this challenge, we reformulate
physical adversarial attacks as a one-shot patch-generation problem. Our
approach generates adversarial patches through a deep generative model that
considers the specific scene context, enabling direct physical deployment in
matching environments. The primary challenge lies in simultaneously achieving
two objectives: generating adversarial patches that effectively mislead object
detection systems while determining contextually appropriate placement within
the scene. We propose MAGIC (Mastering Physical Adversarial Generation In
Context), a novel framework powered by multi-modal LLM agents to address these
challenges. MAGIC automatically understands scene context and orchestrates
adversarial patch generation through the synergistic interaction of language
and vision capabilities. MAGIC orchestrates three specialized LLM agents: The
adv-patch generation agent (GAgent) masters the creation of deceptive patches
through strategic prompt engineering for text-to-image models. The adv-patch
deployment agent (DAgent) ensures contextual coherence by determining optimal
placement strategies based on scene understanding. The self-examination agent
(EAgent) completes this trilogy by providing critical oversight and iterative
refinement of both processes. We validate our method on both digital and
physical level, \ie, nuImage and manually captured real scenes, where both
statistical and visual results prove that our MAGIC is powerful and effectively
for attacking wide-used object detection systems.


http://arxiv.org/abs/2412.07672
FlexLLM: Exploring LLM Customization for Moving Target Defense on Black-Box LLMs Against Jailbreak Attacks. (81%)
Bocheng Chen; Hanqing Guo; Qiben Yan
  Defense in large language models (LLMs) is crucial to counter the numerous
attackers exploiting these systems to generate harmful content through
manipulated prompts, known as jailbreak attacks. Although many defense
strategies have been proposed, they often require access to the model's
internal structure or need additional training, which is impractical for
service providers using LLM APIs, such as OpenAI APIs or Claude APIs. In this
paper, we propose a moving target defense approach that alters decoding
hyperparameters to enhance model robustness against various jailbreak attacks.
Our approach does not require access to the model's internal structure and
incurs no additional training costs. The proposed defense includes two key
components: (1) optimizing the decoding strategy by identifying and adjusting
decoding hyperparameters that influence token generation probabilities, and (2)
transforming the decoding hyperparameters and model system prompts into dynamic
targets, which are continuously altered during each runtime. By continuously
modifying decoding strategies and prompts, the defense effectively mitigates
the existing attacks. Our results demonstrate that our defense is the most
effective against jailbreak attacks in three of the models tested when using
LLMs as black-box APIs. Moreover, our defense offers lower inference costs and
maintains comparable response quality, making it a potential layer of
protection when used alongside other defense methods.


http://arxiv.org/abs/2412.07511
Stealthy and Robust Backdoor Attack against 3D Point Clouds through Additional Point Features. (76%)
Xiaoyang Ning; Qing Xie; Jinyu Xu; Wenbo Jiang; Jiachen Li; Yanchun Ma
  Recently, 3D backdoor attacks have posed a substantial threat to 3D Deep
Neural Networks (3D DNNs) designed for 3D point clouds, which are extensively
deployed in various security-critical applications. Although the existing 3D
backdoor attacks achieved high attack performance, they remain vulnerable to
preprocessing-based defenses (e.g., outlier removal and rotation augmentation)
and are prone to detection by human inspection. In pursuit of a more
challenging-to-defend and stealthy 3D backdoor attack, this paper introduces
the Stealthy and Robust Backdoor Attack (SRBA), which ensures robustness and
stealthiness through intentional design considerations. The key insight of our
attack involves applying a uniform shift to the additional point features of
point clouds (e.g., reflection intensity) widely utilized as part of inputs for
3D DNNs as the trigger. Without altering the geometric information of the point
clouds, our attack ensures visual consistency between poisoned and benign
samples, and demonstrate robustness against preprocessing-based defenses. In
addition, to automate our attack, we employ Bayesian Optimization (BO) to
identify the suitable trigger. Extensive experiments suggest that SRBA achieves
an attack success rate (ASR) exceeding 94% in all cases, and significantly
outperforms previous SOTA methods when multiple preprocessing operations are
applied during training.


http://arxiv.org/abs/2412.07231
Adversarial Filtering Based Evasion and Backdoor Attacks to EEG-Based Brain-Computer Interfaces. (68%)
Lubin Meng; Xue Jiang; Xiaoqing Chen; Wenzhong Liu; Hanbin Luo; Dongrui Wu
  A brain-computer interface (BCI) enables direct communication between the
brain and an external device. Electroencephalogram (EEG) is a common input
signal for BCIs, due to its convenience and low cost. Most research on
EEG-based BCIs focuses on the accurate decoding of EEG signals, while ignoring
their security. Recent studies have shown that machine learning models in BCIs
are vulnerable to adversarial attacks. This paper proposes adversarial
filtering based evasion and backdoor attacks to EEG-based BCIs, which are very
easy to implement. Experiments on three datasets from different BCI paradigms
demonstrated the effectiveness of our proposed attack approaches. To our
knowledge, this is the first study on adversarial filtering for EEG-based BCIs,
raising a new security concern and calling for more attention on the security
of BCIs.


http://arxiv.org/abs/2412.07199
A Parametric Approach to Adversarial Augmentation for Cross-Domain Iris Presentation Attack Detection. (61%)
Debasmita Pal; Redwan Sony; Arun Ross
  Iris-based biometric systems are vulnerable to presentation attacks (PAs),
where adversaries present physical artifacts (e.g., printed iris images,
textured contact lenses) to defeat the system. This has led to the development
of various presentation attack detection (PAD) algorithms, which typically
perform well in intra-domain settings. However, they often struggle to
generalize effectively in cross-domain scenarios, where training and testing
employ different sensors, PA instruments, and datasets. In this work, we use
adversarial training samples of both bonafide irides and PAs to improve the
cross-domain performance of a PAD classifier. The novelty of our approach lies
in leveraging transformation parameters from classical data augmentation
schemes (e.g., translation, rotation) to generate adversarial samples. We
achieve this through a convolutional autoencoder, ADV-GEN, that inputs original
training samples along with a set of geometric and photometric transformations.
The transformation parameters act as regularization variables, guiding ADV-GEN
to generate adversarial samples in a constrained search space. Experiments
conducted on the LivDet-Iris 2017 database, comprising four datasets, and the
LivDet-Iris 2020 dataset, demonstrate the efficacy of our proposed method. The
code is available at https://github.com/iPRoBe-lab/ADV-GEN-IrisPAD.


http://arxiv.org/abs/2412.12145
Na'vi or Knave: Jailbreaking Language Models via Metaphorical Avatars. (50%)
Yu Yan; Sheng Sun; Junqi Tong; Min Liu; Qi Li
  Metaphor serves as an implicit approach to convey information, while enabling
the generalized comprehension of complex subjects. However, metaphor can
potentially be exploited to bypass the safety alignment mechanisms of Large
Language Models (LLMs), leading to the theft of harmful knowledge. In our
study, we introduce a novel attack framework that exploits the imaginative
capacity of LLMs to achieve jailbreaking, the J\underline{\textbf{A}}ilbreak
\underline{\textbf{V}}ia \underline{\textbf{A}}dversarial
Me\underline{\textbf{TA}} -pho\underline{\textbf{R}} (\textit{AVATAR}).
Specifically, to elicit the harmful response, AVATAR extracts harmful entities
from a given harmful target and maps them to innocuous adversarial entities
based on LLM's imagination. Then, according to these metaphors, the harmful
target is nested within human-like interaction for jailbreaking adaptively.
Experimental results demonstrate that AVATAR can effectively and transferablly
jailbreak LLMs and achieve a state-of-the-art attack success rate across
multiple advanced LLMs. Our study exposes a security risk in LLMs from their
endogenous imaginative capabilities. Furthermore, the analytical study reveals
the vulnerability of LLM to adversarial metaphors and the necessity of
developing defense methods against jailbreaking caused by the adversarial
metaphor. \textcolor{orange}{ \textbf{Warning: This paper contains potentially
harmful content from LLMs.}}


http://arxiv.org/abs/2412.07253
CapGen:An Environment-Adaptive Generator of Adversarial Patches. (13%)
Chaoqun Li; Zhuodong Liu; Huanqian Yan; Hang Su
  Adversarial patches, often used to provide physical stealth protection for
critical assets and assess perception algorithm robustness, usually neglect the
need for visual harmony with the background environment, making them easily
noticeable. Moreover, existing methods primarily concentrate on improving
attack performance, disregarding the intricate dynamics of adversarial patch
elements. In this work, we introduce the Camouflaged Adversarial Pattern
Generator (CAPGen), a novel approach that leverages specific base colors from
the surrounding environment to produce patches that seamlessly blend with their
background for superior visual stealthiness while maintaining robust
adversarial performance. We delve into the influence of both patterns (i.e.,
color-agnostic texture information) and colors on the effectiveness of attacks
facilitated by patches, discovering that patterns exert a more pronounced
effect on performance than colors. Based on these findings, we propose a rapid
generation strategy for adversarial patches. This involves updating the colors
of high-performance adversarial patches to align with those of the new
environment, ensuring visual stealthiness without compromising adversarial
impact. This paper is the first to comprehensively examine the roles played by
patterns and colors in the context of adversarial patches.


http://arxiv.org/abs/2412.07192
PrisonBreak: Jailbreaking Large Language Models with Fewer Than Twenty-Five Targeted Bit-flips. (9%)
Zachary Coalson; Jeonghyun Woo; Shiyang Chen; Yu Sun; Lishan Yang; Prashant Nair; Bo Fang; Sanghyun Hong
  We introduce a new class of attacks on commercial-scale (human-aligned)
language models that induce jailbreaking through targeted bitwise corruptions
in model parameters. Our adversary can jailbreak billion-parameter language
models with fewer than 25 bit-flips in all cases$-$and as few as 5 in
some$-$using up to 40$\times$ less bit-flips than existing attacks on computer
vision models at least 100$\times$ smaller. Unlike prompt-based jailbreaks, our
attack renders these models in memory 'uncensored' at runtime, allowing them to
generate harmful responses without any input modifications. Our attack
algorithm efficiently identifies target bits to flip, offering up to 20$\times$
more computational efficiency than previous methods. This makes it practical
for language models with billions of parameters. We show an end-to-end
exploitation of our attack using software-induced fault injection, Rowhammer
(RH). Our work examines 56 DRAM RH profiles from DDR4 and LPDDR4X devices with
different RH vulnerabilities. We show that our attack can reliably induce
jailbreaking in systems similar to those affected by prior bit-flip attacks.
Moreover, our approach remains effective even against highly RH-secure systems
(e.g., 46$\times$ more secure than previously tested systems). Our analyses
further reveal that: (1) models with less post-training alignment require fewer
bit flips to jailbreak; (2) certain model components, such as value projection
layers, are substantially more vulnerable than others; and (3) our method is
mechanistically different than existing jailbreaks. Our findings highlight a
pressing, practical threat to the language model ecosystem and underscore the
need for research to protect these models from bit-flip attacks.


http://arxiv.org/abs/2412.07249
Buster: Implanting Semantic Backdoor into Text Encoder to Mitigate NSFW Content Generation. (2%)
Xin Zhao; Xiaojun Chen; Yuexin Xuan; Zhendong Zhao; Xiaojun Jia; Xinfeng Li; Xiaofeng Wang
  The rise of deep learning models in the digital era has raised substantial
concerns regarding the generation of Not-Safe-for-Work (NSFW) content. Existing
defense methods primarily involve model fine-tuning and post-hoc content
moderation. Nevertheless, these approaches largely lack scalability in
eliminating harmful content, degrade the quality of benign image generation, or
incur high inference costs. To address these challenges, we propose an
innovative framework named \textit{Buster}, which injects backdoors into the
text encoder to prevent NSFW content generation. Buster leverages deep semantic
information rather than explicit prompts as triggers, redirecting NSFW prompts
towards targeted benign prompts. Additionally, Buster employs energy-based
training data generation through Langevin dynamics for adversarial knowledge
augmentation, thereby ensuring robustness in harmful concept definition. This
approach demonstrates exceptional resilience and scalability in mitigating NSFW
content. Particularly, Buster fine-tunes the text encoder of Text-to-Image
models within merely five minutes, showcasing its efficiency. Our extensive
experiments denote that Buster outperforms nine state-of-the-art baselines,
achieving a superior NSFW content removal rate of at least 91.2\% while
preserving the quality of harmless images.


http://arxiv.org/abs/2412.06727
Take Fake as Real: Realistic-like Robust Black-box Adversarial Attack to Evade AIGC Detection. (99%)
Caiyun Xie; Dengpan Ye; Yunming Zhang; Long Tang; Yunna Lv; Jiacheng Deng; Jiawei Song
  The security of AI-generated content (AIGC) detection is crucial for ensuring
multimedia content credibility. To enhance detector security, research on
adversarial attacks has become essential. However, most existing adversarial
attacks focus only on GAN-generated facial images detection, struggle to be
effective on multi-class natural images and diffusion-based detectors, and
exhibit poor invisibility. To fill this gap, we first conduct an in-depth
analysis of the vulnerability of AIGC detectors and discover the feature that
detectors vary in vulnerability to different post-processing. Then, considering
that the detector is agnostic in real-world scenarios and given this discovery,
we propose a Realistic-like Robust Black-box Adversarial attack (R$^2$BA) with
post-processing fusion optimization. Unlike typical perturbations, R$^2$BA uses
real-world post-processing, i.e., Gaussian blur, JPEG compression, Gaussian
noise and light spot to generate adversarial examples. Specifically, we use a
stochastic particle swarm algorithm with inertia decay to optimize
post-processing fusion intensity and explore the detector's decision boundary.
Guided by the detector's fake probability, R$^2$BA enhances/weakens the
detector-vulnerable/detector-robust post-processing intensity to strike a
balance between adversariality and invisibility. Extensive experiments on
popular/commercial AIGC detectors and datasets demonstrate that R$^2$BA
exhibits impressive anti-detection performance, excellent invisibility, and
strong robustness in GAN-based and diffusion-based cases. Compared to
state-of-the-art white-box and black-box attacks, R$^2$BA shows significant
improvements of 15\%--72\% and 21\%--47\% in anti-detection performance under
the original and robust scenario respectively, offering valuable insights for
the security of AIGC detection in real-world applications.


http://arxiv.org/abs/2412.07078
Defensive Dual Masking for Robust Adversarial Defense. (99%)
Wangli Yang; Jie Yang; Yi Guo; Johan Barthelemy
  The field of textual adversarial defenses has gained considerable attention
in recent years due to the increasing vulnerability of natural language
processing (NLP) models to adversarial attacks, which exploit subtle
perturbations in input text to deceive models. This paper introduces the
Defensive Dual Masking (DDM) algorithm, a novel approach designed to enhance
model robustness against such attacks. DDM utilizes a unique adversarial
training strategy where [MASK] tokens are strategically inserted into training
samples to prepare the model to handle adversarial perturbations more
effectively. During inference, potentially adversarial tokens are dynamically
replaced with [MASK] tokens to neutralize potential threats while preserving
the core semantics of the input. The theoretical foundation of our approach is
explored, demonstrating how the selective masking mechanism strengthens the
model's ability to identify and mitigate adversarial manipulations. Our
empirical evaluation across a diverse set of benchmark datasets and attack
mechanisms consistently shows that DDM outperforms state-of-the-art defense
techniques, improving model accuracy and robustness. Moreover, when applied to
Large Language Models (LLMs), DDM also enhances their resilience to adversarial
attacks, providing a scalable defense mechanism for large-scale NLP
applications.


http://arxiv.org/abs/2412.06215
A Real-Time Defense Against Object Vanishing Adversarial Patch Attacks for Object Detection in Autonomous Vehicles. (97%)
Jaden Mu
  Autonomous vehicles (AVs) increasingly use DNN-based object detection models
in vision-based perception. Correct detection and classification of obstacles
is critical to ensure safe, trustworthy driving decisions. Adversarial patches
aim to fool a DNN with intentionally generated patterns concentrated in a
localized region of an image. In particular, object vanishing patch attacks can
cause object detection models to fail to detect most or all objects in a scene,
posing a significant practical threat to AVs.
  This work proposes ADAV (Adversarial Defense for Autonomous Vehicles), a
novel defense methodology against object vanishing patch attacks specifically
designed for autonomous vehicles. Unlike existing defense methods which have
high latency or are designed for static images, ADAV runs in real-time and
leverages contextual information from prior frames in an AV's video feed. ADAV
checks if the object detector's output for the target frame is temporally
consistent with the output from a previous reference frame to detect the
presence of a patch. If the presence of a patch is detected, ADAV uses
gradient-based attribution to localize adversarial pixels that break temporal
consistency. This two stage procedure allows ADAV to efficiently process clean
inputs, and both stages are optimized to be low latency. ADAV is evaluated
using real-world driving data from the Berkeley Deep Drive BDD100K dataset, and
demonstrates high adversarial and clean performance.


http://arxiv.org/abs/2412.06219
Data Free Backdoor Attacks. (64%)
Bochuan Cao; Jinyuan Jia; Chuxuan Hu; Wenbo Guo; Zhen Xiang; Jinghui Chen; Bo Li; Dawn Song
  Backdoor attacks aim to inject a backdoor into a classifier such that it
predicts any input with an attacker-chosen backdoor trigger as an
attacker-chosen target class. Existing backdoor attacks require either
retraining the classifier with some clean data or modifying the model's
architecture. As a result, they are 1) not applicable when clean data is
unavailable, 2) less efficient when the model is large, and 3) less stealthy
due to architecture changes. In this work, we propose DFBA, a novel
retraining-free and data-free backdoor attack without changing the model
architecture. Technically, our proposed method modifies a few parameters of a
classifier to inject a backdoor. Through theoretical analysis, we verify that
our injected backdoor is provably undetectable and unremovable by various
state-of-the-art defenses under mild assumptions. Our evaluation on multiple
datasets further demonstrates that our injected backdoor: 1) incurs negligible
classification loss, 2) achieves 100% attack success rates, and 3) bypasses six
existing state-of-the-art defenses. Moreover, our comparison with a
state-of-the-art non-data-free backdoor attack shows our attack is more
stealthy and effective against various defenses while achieving less
classification accuracy loss.


http://arxiv.org/abs/2412.07097
On Evaluating the Durability of Safeguards for Open-Weight LLMs. (38%)
Xiangyu Qi; Boyi Wei; Nicholas Carlini; Yangsibo Huang; Tinghao Xie; Luxi He; Matthew Jagielski; Milad Nasr; Prateek Mittal; Peter Henderson
  Stakeholders -- from model developers to policymakers -- seek to minimize the
dual-use risks of large language models (LLMs). An open challenge to this goal
is whether technical safeguards can impede the misuse of LLMs, even when models
are customizable via fine-tuning or when model weights are fully open. In
response, several recent studies have proposed methods to produce durable LLM
safeguards for open-weight LLMs that can withstand adversarial modifications of
the model's weights via fine-tuning. This holds the promise of raising
adversaries' costs even under strong threat models where adversaries can
directly fine-tune model weights. However, in this paper, we urge for more
careful characterization of the limits of these approaches. Through several
case studies, we demonstrate that even evaluating these defenses is exceedingly
difficult and can easily mislead audiences into thinking that safeguards are
more durable than they really are. We draw lessons from the evaluation pitfalls
that we identify and suggest future research carefully cabin claims to more
constrained, well-defined, and rigorously examined threat models, which can
provide more useful and candid assessments to stakeholders.


http://arxiv.org/abs/2412.06966
Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy, Research, and Practice. (3%)
A. Feder Cooper; Christopher A. Choquette-Choo; Miranda Bogen; Matthew Jagielski; Katja Filippova; Ken Ziyu Liu; Alexandra Chouldechova; Jamie Hayes; Yangsibo Huang; Niloofar Mireshghallah; Ilia Shumailov; Eleni Triantafillou; Peter Kairouz; Nicole Mitchell; Percy Liang; Daniel E. Ho; Yejin Choi; Sanmi Koyejo; Fernando Delgado; James Grimmelmann; Vitaly Shmatikov; Sa Christopher De; Solon Barocas; Amy Cyphert; Mark Lemley; danah boyd; Jennifer Wortman Vaughan; Miles Brundage; David Bau; Seth Neel; Abigail Z. Jacobs; Andreas Terzis; Hanna Wallach; Nicolas Papernot; Katherine Lee
  We articulate fundamental mismatches between technical methods for machine
unlearning in Generative AI, and documented aspirations for broader impact that
these methods could have for law and policy. These aspirations are both
numerous and varied, motivated by issues that pertain to privacy, copyright,
safety, and more. For example, unlearning is often invoked as a solution for
removing the effects of targeted information from a generative-AI model's
parameters, e.g., a particular individual's personal data or in-copyright
expression of Spiderman that was included in the model's training data.
Unlearning is also proposed as a way to prevent a model from generating
targeted types of information in its outputs, e.g., generations that closely
resemble a particular individual's data or reflect the concept of "Spiderman."
Both of these goals--the targeted removal of information from a model and the
targeted suppression of information from a model's outputs--present various
technical and substantive challenges. We provide a framework for thinking
rigorously about these challenges, which enables us to be clear about why
unlearning is not a general-purpose solution for circumscribing generative-AI
model behavior in service of broader positive impact. We aim for conceptual
clarity and to encourage more thoughtful communication among machine learning
(ML), law, and policy experts who seek to develop and apply technical methods
for compliance with policy objectives.


http://arxiv.org/abs/2412.07129
StyleMark: A Robust Watermarking Method for Art Style Images Against Black-Box Arbitrary Style Transfer. (2%)
Yunming Zhang; Dengpan Ye; Sipeng Shen; Jun Wang
  Arbitrary Style Transfer (AST) achieves the rendering of real natural images
into the painting styles of arbitrary art style images, promoting art
communication. However, misuse of unauthorized art style images for AST may
infringe on artists' copyrights. One countermeasure is robust watermarking,
which tracks image propagation by embedding copyright watermarks into carriers.
Unfortunately, AST-generated images lose the structural and semantic
information of the original style image, hindering end-to-end robust tracking
by watermarks. To fill this gap, we propose StyleMark, the first robust
watermarking method for black-box AST, which can be seamlessly applied to art
style images achieving precise attribution of artistic styles after AST.
Specifically, we propose a new style watermark network that adjusts the mean
activations of style features through multi-scale watermark embedding, thereby
planting watermark traces into the shared style feature space of style images.
Furthermore, we design a distribution squeeze loss, which constrain content
statistical feature distortion, forcing the reconstruction network to focus on
integrating style features with watermarks, thus optimizing the intrinsic
watermark distribution. Finally, based on solid end-to-end training, StyleMark
mitigates the optimization conflict between robustness and watermark
invisibility through decoder fine-tuning under random noise. Experimental
results demonstrate that StyleMark exhibits significant robustness against
black-box AST and common pixel-level distortions, while also securely defending
against malicious adaptive attacks.


http://arxiv.org/abs/2412.07003
Understanding Gradient Descent through the Training Jacobian. (1%)
Nora Belrose; Adam Scherlis
  We examine the geometry of neural network training using the Jacobian of
trained network parameters with respect to their initial values. Our analysis
reveals low-dimensional structure in the training process which is dependent on
the input data but largely independent of the labels. We find that the singular
value spectrum of the Jacobian matrix consists of three distinctive regions: a
"chaotic" region of values orders of magnitude greater than one, a large "bulk"
region of values extremely close to one, and a "stable" region of values less
than one. Along each bulk direction, the left and right singular vectors are
nearly identical, indicating that perturbations to the initialization are
carried through training almost unchanged. These perturbations have virtually
no effect on the network's output in-distribution, yet do have an effect far
out-of-distribution. While the Jacobian applies only locally around a single
initialization, we find substantial overlap in bulk subspaces for different
random seeds.


http://arxiv.org/abs/2412.06556
Vulnerability, Where Art Thou? An Investigation of Vulnerability Management in Android Smartphone Chipsets. (1%)
Daniel Klischies; Philipp Mackensen; Veelasha Moonsamy
  Vulnerabilities in Android smartphone chipsets have severe consequences, as
recent real-world attacks have demonstrated that adversaries can leverage
vulnerabilities to execute arbitrary code or exfiltrate confidential
information. Despite the far-reaching impact of such attacks, the lifecycle of
chipset vulnerabilities has yet to be investigated, with existing papers
primarily investigating vulnerabilities in the Android operating system. This
paper provides a comprehensive and empirical study of the current state of
smartphone chipset vulnerability management within the Android ecosystem. For
the first time, we create a unified knowledge base of 3,676 chipset
vulnerabilities affecting 437 chipset models from all four major chipset
manufacturers, combined with 6,866 smartphone models. Our analysis revealed
that the same vulnerabilities are often included in multiple generations of
chipsets, providing novel empirical evidence that vulnerabilities are inherited
through multiple chipset generations. Furthermore, we demonstrate that the
commonly accepted 90-day responsible vulnerability disclosure period is seldom
adhered to. We find that a single vulnerability often affects hundreds to
thousands of different smartphone models, for which update availability is, as
we show, often unclear or heavily delayed. Leveraging the new insights gained
from our empirical analysis, we recommend several changes that chipset
manufacturers can implement to improve the security posture of their products.
At the same time, our knowledge base enables academic researchers to conduct
more representative evaluations of smartphone chipsets, accurately assess the
impact of vulnerabilities they discover, and identify avenues for future
research.


http://arxiv.org/abs/2412.05943
Adversarial Transferability in Deep Denoising Models: Theoretical Insights and Robustness Enhancement via Out-of-Distribution Typical Set Sampling. (99%)
Jie Ning; Jiebao Sun; Shengzhu Shi; Zhichang Guo; Yao Li; Hongwei Li; Boying Wu
  Deep learning-based image denoising models demonstrate remarkable
performance, but their lack of robustness analysis remains a significant
concern. A major issue is that these models are susceptible to adversarial
attacks, where small, carefully crafted perturbations to input data can cause
them to fail. Surprisingly, perturbations specifically crafted for one model
can easily transfer across various models, including CNNs, Transformers,
unfolding models, and plug-and-play models, leading to failures in those models
as well. Such high adversarial transferability is not observed in
classification models. We analyze the possible underlying reasons behind the
high adversarial transferability through a series of hypotheses and validation
experiments. By characterizing the manifolds of Gaussian noise and adversarial
perturbations using the concept of typical set and the asymptotic equipartition
property, we prove that adversarial samples deviate slightly from the typical
set of the original input distribution, causing the models to fail. Based on
these insights, we propose a novel adversarial defense method: the
Out-of-Distribution Typical Set Sampling Training strategy (TS). TS not only
significantly enhances the model's robustness but also marginally improves
denoising performance compared to the original model.


http://arxiv.org/abs/2412.06149
An Effective and Resilient Backdoor Attack Framework against Deep Neural Networks and Vision Transformers. (89%)
Xueluan Gong; Bowei Tian; Meng Xue; Yuan Wu; Yanjiao Chen; Qian Wang
  Recent studies have revealed the vulnerability of Deep Neural Network (DNN)
models to backdoor attacks. However, existing backdoor attacks arbitrarily set
the trigger mask or use a randomly selected trigger, which restricts the
effectiveness and robustness of the generated backdoor triggers. In this paper,
we propose a novel attention-based mask generation methodology that searches
for the optimal trigger shape and location. We also introduce a
Quality-of-Experience (QoE) term into the loss function and carefully adjust
the transparency value of the trigger in order to make the backdoored samples
to be more natural. To further improve the prediction accuracy of the victim
model, we propose an alternating retraining algorithm in the backdoor injection
process. The victim model is retrained with mixed poisoned datasets in even
iterations and with only benign samples in odd iterations. Besides, we launch
the backdoor attack under a co-optimized attack framework that alternately
optimizes the backdoor trigger and backdoored model to further improve the
attack performance. Apart from DNN models, we also extend our proposed attack
method against vision transformers. We evaluate our proposed method with
extensive experiments on VGG-Flower, CIFAR-10, GTSRB, CIFAR-100, and ImageNette
datasets. It is shown that we can increase the attack success rate by as much
as 82\% over baselines when the poison ratio is low and achieve a high QoE of
the backdoored samples. Our proposed backdoor attack framework also showcases
robustness against state-of-the-art backdoor defenses.


http://arxiv.org/abs/2412.05883
Understanding the Impact of Graph Reduction on Adversarial Robustness in Graph Neural Networks. (78%)
Kerui Wu; Ka-Ho Chow; Wenqi Wei; Lei Yu
  As Graph Neural Networks (GNNs) become increasingly popular for learning from
large-scale graph data across various domains, their susceptibility to
adversarial attacks when using graph reduction techniques for scalability
remains underexplored. In this paper, we present an extensive empirical study
to investigate the impact of graph reduction techniques, specifically graph
coarsening and sparsification, on the robustness of GNNs against adversarial
attacks. Through extensive experiments involving multiple datasets and GNN
architectures, we examine the effects of four sparsification and six coarsening
methods on the poisoning attacks. Our results indicate that, while graph
sparsification can mitigate the effectiveness of certain poisoning attacks,
such as Mettack, it has limited impact on others, like PGD. Conversely, graph
coarsening tends to amplify the adversarial impact, significantly reducing
classification accuracy as the reduction ratio decreases. Additionally, we
provide a novel analysis of the causes driving these effects and examine how
defensive GNN models perform under graph reduction, offering practical insights
for designing robust GNNs within graph acceleration systems.


http://arxiv.org/abs/2412.05980
Anti-Reference: Universal and Immediate Defense Against Reference-Based Generation. (22%)
Yiren Song; Shengtao Lou; Xiaokang Liu; Hai Ci; Pei Yang; Jiaming Liu; Mike Zheng Shou
  Diffusion models have revolutionized generative modeling with their
exceptional ability to produce high-fidelity images. However, misuse of such
potent tools can lead to the creation of fake news or disturbing content
targeting individuals, resulting in significant social harm. In this paper, we
introduce Anti-Reference, a novel method that protects images from the threats
posed by reference-based generation techniques by adding imperceptible
adversarial noise to the images. We propose a unified loss function that
enables joint attacks on fine-tuning-based customization methods,
non-fine-tuning customization methods, and human-centric driving methods. Based
on this loss, we train a Adversarial Noise Encoder to predict the noise or
directly optimize the noise using the PGD method. Our method shows certain
transfer attack capabilities, effectively challenging both gray-box models and
some commercial APIs. Extensive experiments validate the performance of
Anti-Reference, establishing a new benchmark in image security.


http://arxiv.org/abs/2412.05892
PBI-Attack: Prior-Guided Bimodal Interactive Black-Box Jailbreak Attack for Toxicity Maximization. (15%)
Ruoxi Cheng; Yizhong Ding; Shuirong Cao; Ranjie Duan; Xiaoshuang Jia; Shaowei Yuan; Zhiqiang Wang; Xiaojun Jia
  Understanding the vulnerabilities of Large Vision Language Models (LVLMs) to
jailbreak attacks is essential for their responsible real-world deployment.
Most previous work requires access to model gradients, or is based on human
knowledge (prompt engineering) to complete jailbreak, and they hardly consider
the interaction of images and text, resulting in inability to jailbreak in
black box scenarios or poor performance. To overcome these limitations, we
propose a Prior-Guided Bimodal Interactive Black-Box Jailbreak Attack for
toxicity maximization, referred to as PBI-Attack. Our method begins by
extracting malicious features from a harmful corpus using an alternative LVLM
and embedding these features into a benign image as prior information.
Subsequently, we enhance these features through bidirectional cross-modal
interaction optimization, which iteratively optimizes the bimodal perturbations
in an alternating manner through greedy search, aiming to maximize the toxicity
of the generated response. The toxicity level is quantified using a
well-trained evaluation model. Experiments demonstrate that PBI-Attack
outperforms previous state-of-the-art jailbreak methods, achieving an average
attack success rate of 92.5% across three open-source LVLMs and around 67.3% on
three closed-source LVLMs. Disclaimer: This paper contains potentially
disturbing and offensive content.


http://arxiv.org/abs/2412.05829
SABER: Model-agnostic Backdoor Attack on Chain-of-Thought in Neural Code Generation. (4%)
Naizhu Jin; Zhong Li; Yinggang Guo; Chao Su; Tian Zhang; Qingkai Zeng
  Recent studies have proposed integrating Chain-of-Thought (CoT) reasoning to
further enhance the reliability of Code Language Models (CLMs) in generating
code, a step-by-step approach that breaks down complex programming tasks into
manageable sub-problems. Advances in this area have introduced CoT models,
specifically designed to integrate CoT reasoning effectively into language
models, achieving notable improvements in code generation. Despite these
advancements, the security of CoT models has not been systematically studied.
In this study, we aim to fill this gap by investigating the vulnerability of
CoT models to backdoor injection in code generation tasks. To address this, we
propose a model-agnostic backdoor attack method SABER
(\textbf{S}elf-\textbf{A}ttention-\textbf{B}as\textbf{E}d backdoo\textbf{R})
based on the self-attention mechanism. SABER begins by selecting a malicious
output as the backdoor using code mutation operations. It then identifies
tokens most relevant to poisoned content by analyzing self-attention scores in
the CodeBERT model. Finally, it applies semantic-preserving perturbations to
generate adaptive and natural triggers. Our experiments on HumanEval-CoT and
OpenEval-CoT test sets demonstrate that CoT models are susceptible to backdoor
attacks via data poisoning. Taking the OpenEval-CoT dataset as an example,
SABER achieves an ASR of 76.19%, representing an improvement of 14.29% over
RIPPLe and a substantial 23.08% enhancement compared to BadPre. Further
evaluations using ONION for automated detection and human studies reveal that
SABER is stealthier and harder to detect, bypassing 77.27% of automated
detection, with a human detection rate of just 3.17%. Our findings reveal that
backdoors can be injected into CoT models to manipulate downstream code
generation tasks.


http://arxiv.org/abs/2412.06181
Enhancing Adversarial Resistance in LLMs with Recursion. (1%)
Bryan Li; Sounak Bagchi; Zizhan Wang
  The increasing integration of Large Language Models (LLMs) into society
necessitates robust defenses against vulnerabilities from jailbreaking and
adversarial prompts. This project proposes a recursive framework for enhancing
the resistance of LLMs to manipulation through the use of prompt simplification
techniques. By increasing the transparency of complex and confusing adversarial
prompts, the proposed method enables more reliable detection and prevention of
malicious inputs. Our findings attempt to address a critical problem in AI
safety and security, providing a foundation for the development of systems able
to distinguish harmless inputs from prompts containing malicious intent. As
LLMs continue to be used in diverse applications, the importance of such
safeguards will only grow.


http://arxiv.org/abs/2412.06157
Membership Inference Attacks and Defenses in Federated Learning: A Survey. (1%)
Li Bai; Haibo Hu; Qingqing Ye; Haoyang Li; Leixia Wang; Jianliang Xu
  Federated learning is a decentralized machine learning approach where clients
train models locally and share model updates to develop a global model. This
enables low-resource devices to collaboratively build a high-quality model
without requiring direct access to the raw training data. However, despite only
sharing model updates, federated learning still faces several privacy
vulnerabilities. One of the key threats is membership inference attacks, which
target clients' privacy by determining whether a specific example is part of
the training set. These attacks can compromise sensitive information in
real-world applications, such as medical diagnoses within a healthcare system.
Although there has been extensive research on membership inference attacks, a
comprehensive and up-to-date survey specifically focused on it within federated
learning is still absent. To fill this gap, we categorize and summarize
membership inference attacks and their corresponding defense strategies based
on their characteristics in this setting. We introduce a unique taxonomy of
existing attack research and provide a systematic overview of various
countermeasures. For these studies, we thoroughly analyze the strengths and
weaknesses of different approaches. Finally, we identify and discuss key future
research directions for readers interested in advancing the field.


http://arxiv.org/abs/2412.05734
PrivAgent: Agentic-based Red-teaming for LLM Privacy Leakage. (92%)
Yuzhou Nie; Zhun Wang; Ye Yu; Xian Wu; Xuandong Zhao; Wenbo Guo; Dawn Song
  Recent studies have discovered that LLMs have serious privacy leakage
concerns, where an LLM may be fooled into outputting private information under
carefully crafted adversarial prompts. These risks include leaking system
prompts, personally identifiable information, training data, and model
parameters. Most existing red-teaming approaches for privacy leakage rely on
humans to craft the adversarial prompts. A few automated methods are proposed
for system prompt extraction, but they cannot be applied to more severe risks
(e.g., training data extraction) and have limited effectiveness even for system
prompt extraction.
  In this paper, we propose PrivAgent, a novel black-box red-teaming framework
for LLM privacy leakage. We formulate different risks as a search problem with
a unified attack goal. Our framework trains an open-source LLM through
reinforcement learning as the attack agent to generate adversarial prompts for
different target models under different risks. We propose a novel reward
function to provide effective and fine-grained rewards for the attack agent.
Finally, we introduce customizations to better fit our general framework to
system prompt extraction and training data extraction. Through extensive
evaluations, we first show that PrivAgent outperforms existing automated
methods in system prompt leakage against six popular LLMs. Notably, our
approach achieves a 100% success rate in extracting system prompts from
real-world applications in OpenAI's GPT Store. We also show PrivAgent's
effectiveness in extracting training data from an open-source LLM with a
success rate of 5.9%. We further demonstrate PrivAgent's effectiveness in
evading the existing guardrail defense and its helpfulness in enabling better
safety alignment. Finally, we validate our customized designs through a
detailed ablation study. We release our code here
https://github.com/rucnyz/RedAgent.


http://arxiv.org/abs/2412.05767
DeMem: Privacy-Enhanced Robust Adversarial Learning via De-Memorization. (76%)
Xiaoyu Luo; Qiongxiu Li
  Adversarial robustness, the ability of a model to withstand manipulated
inputs that cause errors, is essential for ensuring the trustworthiness of
machine learning models in real-world applications. However, previous studies
have shown that enhancing adversarial robustness through adversarial training
increases vulnerability to privacy attacks. While differential privacy can
mitigate these attacks, it often compromises robustness against both natural
and adversarial samples. Our analysis reveals that differential privacy
disproportionately impacts low-risk samples, causing an unintended performance
drop. To address this, we propose DeMem, which selectively targets high-risk
samples, achieving a better balance between privacy protection and model
robustness. DeMem is versatile and can be seamlessly integrated into various
adversarial training techniques. Extensive evaluations across multiple training
methods and datasets demonstrate that DeMem significantly reduces privacy
leakage while maintaining robustness against both natural and adversarial
samples. These results confirm DeMem's effectiveness and broad applicability in
enhancing privacy without compromising robustness.


http://arxiv.org/abs/2412.05592
From Flexibility to Manipulation: The Slippery Slope of XAI Evaluation. (47%)
Kristoffer Wickstrøm; Marina Marie-Claire Höhne; Anna Hedström
  The lack of ground truth explanation labels is a fundamental challenge for
quantitative evaluation in explainable artificial intelligence (XAI). This
challenge becomes especially problematic when evaluation methods have numerous
hyperparameters that must be specified by the user, as there is no ground truth
to determine an optimal hyperparameter selection. It is typically not feasible
to do an exhaustive search of hyperparameters so researchers typically make a
normative choice based on similar studies in the literature, which provides
great flexibility for the user. In this work, we illustrate how this
flexibility can be exploited to manipulate the evaluation outcome. We frame
this manipulation as an adversarial attack on the evaluation where seemingly
innocent changes in hyperparameter setting significantly influence the
evaluation outcome. We demonstrate the effectiveness of our manipulation across
several datasets with large changes in evaluation outcomes across several
explanation methods and models. Lastly, we propose a mitigation strategy based
on ranking across hyperparameters that aims to provide robustness towards such
manipulation. This work highlights the difficulty of conducting reliable XAI
evaluation and emphasizes the importance of a holistic and transparent approach
to evaluation in XAI.


http://arxiv.org/abs/2412.05676
Nearly Solved? Robust Deepfake Detection Requires More than Visual Forensics. (33%)
Guy Levy; Nathan Liebmann
  Deepfakes are on the rise, with increased sophistication and prevalence
allowing for high-profile social engineering attacks. Detecting them in the
wild is therefore important as ever, giving rise to new approaches breaking
benchmark records in this task. In line with previous work, we show that
recently developed state-of-the-art detectors are susceptible to classical
adversarial attacks, even in a highly-realistic black-box setting, putting
their usability in question. We argue that crucial 'robust features' of
deepfakes are in their higher semantics, and follow that with evidence that a
detector based on a semantic embedding model is less susceptible to black-box
perturbation attacks. We show that large visuo-lingual models like GPT-4o can
perform zero-shot deepfake detection better than current state-of-the-art
methods, and introduce a novel attack based on high-level semantic
manipulation. Finally, we argue that hybridising low- and high-level detectors
can improve adversarial robustness, based on their complementary strengths and
weaknesses.


http://arxiv.org/abs/2412.05538
Uncovering Vision Modality Threats in Image-to-Image Tasks. (8%)
Hao Cheng; Erjia Xiao; Jiayan Yang; Jiahang Cao; Qiang Zhang; Jize Zhang; Kaidi Xu; Jindong Gu; Renjing Xu
  Current image generation models can effortlessly produce high-quality, highly
realistic images, but this also increases the risk of misuse. In various
Text-to-Image or Image-to-Image tasks, attackers can generate a series of
images containing inappropriate content by simply editing the language modality
input. Currently, to prevent this security threat, the various guard or defense
methods that are proposed also focus on defending the language modality.
However, in practical applications, threats in the visual modality,
particularly in tasks involving the editing of real-world images, pose greater
security risks as they can easily infringe upon the rights of the image owner.
Therefore, this paper uses a method named typographic attack to reveal that
various image generation models also commonly face threats in the vision
modality. Furthermore, we also evaluate the defense performance of various
existing methods when facing threats in the vision modality and uncover their
ineffectiveness. Finally, we propose the Vision Modal Threats in Image
Generation Models (VMT-IGMs) dataset, which would serve as a baseline for
evaluating the vision modality vulnerability of various image generation
models.


http://arxiv.org/abs/2412.05351
Towards Predicting the Success of Transfer-based Attacks by Quantifying Shared Feature Representations. (2%)
Ashley S. Dale; Mei Qiu; Foo Bin Che; Thomas Bsaibes; Lauren Christopher; Paul Salama
  Much effort has been made to explain and improve the success of
transfer-based attacks (TBA) on black-box computer vision models. This work
provides the first attempt at a priori prediction of attack success by
identifying the presence of vulnerable features within target models. Recent
work by Chen and Liu (2024) proposed the manifold attack model, a unifying
framework proposing that successful TBA exist in a common manifold space. Our
work experimentally tests the common manifold space hypothesis by a new
methodology: first, projecting feature vectors from surrogate and target
feature extractors trained on ImageNet onto the same low-dimensional manifold;
second, quantifying any observed structure similarities on the manifold; and
finally, by relating these observed similarities to the success of the TBA. We
find that shared feature representation moderately correlates with increased
success of TBA (\r{ho}= 0.56). This method may be used to predict whether an
attack will transfer without information of the model weights, training,
architecture or details of the attack. The results confirm the presence of
shared feature representations between two feature extractors of different
sizes and complexities, and demonstrate the utility of datasets from different
target domains as test signals for interpreting black-box feature
representations.


http://arxiv.org/abs/2412.05010
Backdooring Outlier Detection Methods: A Novel Attack Approach. (2%)
ZeinabSadat Taghavi; Hossein Mirzaei
  There have been several efforts in backdoor attacks, but these have primarily
focused on the closed-set performance of classifiers (i.e., classification).
This has left a gap in addressing the threat to classifiers' open-set
performance, referred to as outlier detection in the literature. Reliable
outlier detection is crucial for deploying classifiers in critical real-world
applications such as autonomous driving and medical image analysis. First, we
show that existing backdoor attacks fall short in affecting the open-set
performance of classifiers, as they have been specifically designed to confuse
intra-closed-set decision boundaries. In contrast, an effective backdoor attack
for outlier detection needs to confuse the decision boundary between the closed
and open sets. Motivated by this, in this study, we propose BATOD, a novel
Backdoor Attack targeting the Outlier Detection task. Specifically, we design
two categories of triggers to shift inlier samples to outliers and vice versa.
We evaluate BATOD using various real-world datasets and demonstrate its
superior ability to degrade the open-set performance of classifiers compared to
previous attacks, both before and after applying defenses.


http://arxiv.org/abs/2412.05232
LIAR: Leveraging Alignment (Best-of-N) to Jailbreak LLMs in Seconds. (1%)
James Beetham; Souradip Chakraborty; Mengdi Wang; Furong Huang; Amrit Singh Bedi; Mubarak Shah
  Many existing jailbreak techniques rely on solving discrete combinatorial
optimization, while more recent approaches involve training LLMs to generate
multiple adversarial prompts. However, both approaches require significant
computational resources to produce even a single adversarial prompt. We
hypothesize that the inefficiency of current approaches stems from an
inadequate characterization of the jailbreak problem. To address this gap, we
formulate the jailbreak problem in terms of alignment. By starting from an
available safety-aligned model, we leverage an unsafe reward to guide the safe
model towards generating unsafe outputs using alignment techniques (e.g.,
reinforcement learning from human feedback), effectively performing
jailbreaking via alignment. We propose a novel jailbreak method called LIAR
(LeveragIng Alignment to jailbReak). To demonstrate the simplicity and
effectiveness of our approach, we employ a best-of-N method to solve the
alignment problem. LIAR offers significant advantages: lower computational
requirements without additional training, fully black-box operation,
competitive attack success rates, and more human-readable prompts. We provide
theoretical insights into the possibility of jailbreaking a safety-aligned
model, revealing inherent vulnerabilities in current alignment strategies for
LLMs. We also provide sub-optimality guarantees for the proposed \algo.
Experimentally, we achieve ASR comparable to the SoTA with a 10x improvement to
perplexity and a Time-to-Attack measured in seconds rather than tens of hours.


http://arxiv.org/abs/2412.04245
Intriguing Properties of Robust Classification. (96%)
Bernd Prach; Christoph H. Lampert
  Despite extensive research since the community learned about adversarial
examples 10 years ago, we still do not know how to train high-accuracy
classifiers that are guaranteed to be robust to small perturbations of their
inputs. Previous works often argued that this might be because no classifier
exists that is robust and accurate at the same time. However, in computer
vision this assumption does not match reality where humans are usually accurate
and robust on most tasks of interest. We offer an alternative explanation and
show that in certain settings robust generalization is only possible with
unrealistically large amounts of data. More precisely we find a setting where a
robust classifier exists, it is easy to learn an accurate classifier, yet it
requires an exponential amount of data to learn a robust classifier. Based on
this theoretical result, we explore how well robust classifiers generalize on
datasets such as CIFAR-10. We come to the conclusion that on this datasets, the
limitation of current robust models also lies in the generalization, and that
they require a lot of data to do well on the test set. We also show that the
problem is not in the expressiveness or generalization capabilities of current
architectures, and that there are low magnitude features in the data which are
useful for non-robust generalization but are not available for robust
classifiers.


http://arxiv.org/abs/2412.04163
On the Lack of Robustness of Binary Function Similarity Systems. (92%)
Gianluca Capozzi; Tong Tang; Jie Wan; Ziqi Yang; Daniele Cono D'Elia; Luna Giuseppe Antonio Di; Lorenzo Cavallaro; Leonardo Querzoni
  Binary function similarity, which often relies on learning-based algorithms
to identify what functions in a pool are most similar to a given query
function, is a sought-after topic in different communities, including machine
learning, software engineering, and security. Its importance stems from the
impact it has in facilitating several crucial tasks, from reverse engineering
and malware analysis to automated vulnerability detection. Whereas recent work
cast light around performance on this long-studied problem, the research
landscape remains largely lackluster in understanding the resiliency of the
state-of-the-art machine learning models against adversarial attacks. As
security requires to reason about adversaries, in this work we assess the
robustness of such models through a simple yet effective black-box greedy
attack, which modifies the topology and the content of the control flow of the
attacked functions. We demonstrate that this attack is successful in
compromising all the models, achieving average attack success rates of 57.06%
and 95.81% depending on the problem settings (targeted and untargeted attacks).
Our findings are insightful: top performance on clean data does not necessarily
relate to top robustness properties, which explicitly highlights
performance-robustness trade-offs one should consider when deploying such
models, calling for further research.


http://arxiv.org/abs/2412.04776
Megatron: Evasive Clean-Label Backdoor Attacks against Vision Transformer. (76%)
Xueluan Gong; Bowei Tian; Meng Xue; Shuike Li; Yanjiao Chen; Qian Wang
  Vision transformers have achieved impressive performance in various
vision-related tasks, but their vulnerability to backdoor attacks is
under-explored. A handful of existing works focus on dirty-label attacks with
wrongly-labeled poisoned training samples, which may fail if a benign model
trainer corrects the labels. In this paper, we propose Megatron, an evasive
clean-label backdoor attack against vision transformers, where the attacker
injects the backdoor without manipulating the data-labeling process. To
generate an effective trigger, we customize two loss terms based on the
attention mechanism used in transformer networks, i.e., latent loss and
attention diffusion loss. The latent loss aligns the last attention layer
between triggered samples and clean samples of the target label. The attention
diffusion loss emphasizes the attention diffusion area that encompasses the
trigger. A theoretical analysis is provided to underpin the rationale behind
the attention diffusion loss. Extensive experiments on CIFAR-10, GTSRB,
CIFAR-100, and Tiny ImageNet demonstrate the effectiveness of Megatron.
Megatron can achieve attack success rates of over 90% even when the position of
the trigger is slightly shifted during testing. Furthermore, Megatron achieves
better evasiveness than baselines regarding both human visual inspection and
defense strategies (i.e., DBAVT, BAVT, Beatrix, TeCo, and SAGE).


http://arxiv.org/abs/2412.03908
Can Targeted Clean-Label Poisoning Attacks Generalize? (13%)
Zhizhen Chen; Subrat Kishore Dutta; Zhengyu Zhao; Chenhao Lin; Chao Shen; Xiao Zhang
  Targeted poisoning attacks aim to compromise the model's prediction on
specific target samples. In a common clean-label setting, they are achieved by
slightly perturbing a subset of training samples given access to those specific
targets. Despite continuous efforts, it remains unexplored whether such attacks
can generalize to unknown variations of those targets. In this paper, we take
the first step to systematically study this generalization problem. Observing
that the widely adopted, cosine similarity-based attack exhibits limited
generalizability, we propose a well-generalizable attack that leverages both
the direction and magnitude of model gradients. In particular, we explore
diverse target variations, such as an object with varied viewpoints and an
animal species with distinct appearances. Extensive experiments across various
generalization scenarios demonstrate that our method consistently achieves the
best attack effectiveness. For example, our method outperforms the cosine
similarity-based attack by 20.95% in attack success rate with similar overall
accuracy, averaged over four models on two image benchmark datasets. The code
is available at https://github.com/jiaangk/generalizable_tcpa


http://arxiv.org/abs/2412.03993
LaserGuider: A Laser Based Physical Backdoor Attack against Deep Neural Networks. (8%)
Yongjie Xu; Guangke Chen; Fu Song; Yuqi Chen
  Backdoor attacks embed hidden associations between triggers and targets in
deep neural networks (DNNs), causing them to predict the target when a trigger
is present while maintaining normal behavior otherwise. Physical backdoor
attacks, which use physical objects as triggers, are feasible but lack remote
control, temporal stealthiness, flexibility, and mobility. To overcome these
limitations, in this work, we propose a new type of backdoor triggers utilizing
lasers that feature long-distance transmission and instant-imaging properties.
Based on the laser-based backdoor triggers, we present a physical backdoor
attack, called LaserGuider, which possesses remote control ability and achieves
high temporal stealthiness, flexibility, and mobility. We also introduce a
systematic approach to optimize laser parameters for improving attack
effectiveness. Our evaluation on traffic sign recognition DNNs, critical in
autonomous vehicles, demonstrates that LaserGuider with three different
laser-based triggers achieves over 90% attack success rate with negligible
impact on normal inputs. Additionally, we release LaserMark, the first dataset
of real world traffic signs stamped with physical laser spots, to support
further research in backdoor attacks and defenses.


http://arxiv.org/abs/2412.03876
Safeguarding Text-to-Image Generation via Inference-Time Prompt-Noise Optimization. (3%)
Jiangweizhi Peng; Zhiwei Tang; Gaowen Liu; Charles Fleming; Mingyi Hong
  Text-to-Image (T2I) diffusion models are widely recognized for their ability
to generate high-quality and diverse images based on text prompts. However,
despite recent advances, these models are still prone to generating unsafe
images containing sensitive or inappropriate content, which can be harmful to
users. Current efforts to prevent inappropriate image generation for diffusion
models are easy to bypass and vulnerable to adversarial attacks. How to ensure
that T2I models align with specific safety goals remains a significant
challenge. In this work, we propose a novel, training-free approach, called
Prompt-Noise Optimization (PNO), to mitigate unsafe image generation. Our
method introduces a novel optimization framework that leverages both the
continuous prompt embedding and the injected noise trajectory in the sampling
process to generate safe images. Extensive numerical results demonstrate that
our framework achieves state-of-the-art performance in suppressing toxic image
generations and demonstrates robustness to adversarial attacks, without needing
to tune the model parameters. Furthermore, compared with existing methods, PNO
uses comparable generation time while offering the best tradeoff between the
conflicting goals of safe generation and prompt-image alignment.


http://arxiv.org/abs/2412.04415
Targeting the Core: A Simple and Effective Method to Attack RAG-based Agents via Direct LLM Manipulation. (2%)
Xuying Li; Zhuo Li; Yuji Kosuga; Yasuhiro Yoshida; Victor Bian
  AI agents, powered by large language models (LLMs), have transformed
human-computer interactions by enabling seamless, natural, and context-aware
communication. While these advancements offer immense utility, they also
inherit and amplify inherent safety risks such as bias, fairness,
hallucinations, privacy breaches, and a lack of transparency. This paper
investigates a critical vulnerability: adversarial attacks targeting the LLM
core within AI agents. Specifically, we test the hypothesis that a deceptively
simple adversarial prefix, such as \textit{Ignore the document}, can compel
LLMs to produce dangerous or unintended outputs by bypassing their contextual
safeguards. Through experimentation, we demonstrate a high attack success rate
(ASR), revealing the fragility of existing LLM defenses. These findings
emphasize the urgent need for robust, multi-layered security measures tailored
to mitigate vulnerabilities at the LLM level and within broader agent-based
architectures.


http://arxiv.org/abs/2412.03235
Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts? (99%)
Sravanti Addepalli; Yerram Varun; Arun Suggala; Karthikeyan Shanmugam; Prateek Jain
  Large Language Models (LLMs) are known to be susceptible to crafted
adversarial attacks or jailbreaks that lead to the generation of objectionable
content despite being aligned to human preferences using safety fine-tuning
methods. While the large dimensionality of input token space makes it
inevitable to find adversarial prompts that can jailbreak these models, we aim
to evaluate whether safety fine-tuned LLMs are safe against natural prompts
which are semantically related to toxic seed prompts that elicit safe responses
after alignment. We surprisingly find that popular aligned LLMs such as GPT-4
can be compromised using naive prompts that are NOT even crafted with an
objective of jailbreaking the model. Furthermore, we empirically show that
given a seed prompt that elicits a toxic response from an unaligned model, one
can systematically generate several semantically related natural prompts that
can jailbreak aligned LLMs. Towards this, we propose a method of Response
Guided Question Augmentation (ReG-QA) to evaluate the generalization of safety
aligned LLMs to natural prompts, that first generates several toxic answers
given a seed question using an unaligned LLM (Q to A), and further leverages an
LLM to generate questions that are likely to produce these answers (A to Q). We
interestingly find that safety fine-tuned LLMs such as GPT-4o are vulnerable to
producing natural jailbreak questions from unsafe content (without denial) and
can thus be used for the latter (A to Q) step. We obtain attack success rates
that are comparable to/ better than leading adversarial attack methods on the
JailbreakBench leaderboard, while being significantly more stable against
defenses such as Smooth-LLM and Synonym Substitution, which are effective
against existing all attacks on the leaderboard.


http://arxiv.org/abs/2412.03539
NODE-AdvGAN: Improving the transferability and perceptual similarity of adversarial examples by dynamic-system-driven adversarial generative model. (99%)
Xinheng Xie; Yue Wu; Cuiyu He
  Understanding adversarial examples is crucial for improving the model's
robustness, as they introduce imperceptible perturbations that deceive models.
Effective adversarial examples, therefore, offer the potential to train more
robust models by removing their singularities. We propose NODE-AdvGAN, a novel
approach that treats adversarial generation as a continuous process and employs
a Neural Ordinary Differential Equation (NODE) for simulating the dynamics of
the generator. By mimicking the iterative nature of traditional gradient-based
methods, NODE-AdvGAN generates smoother and more precise perturbations that
preserve high perceptual similarity when added to benign images. We also
propose a new training strategy, NODE-AdvGAN-T, which enhances transferability
in black-box attacks by effectively tuning noise parameters during training.
Experiments demonstrate that NODE-AdvGAN and NODE-AdvGAN-T generate more
effective adversarial examples that achieve higher attack success rates while
preserving better perceptual quality than traditional GAN-based methods.


http://arxiv.org/abs/2412.03051
Less is More: A Stealthy and Efficient Adversarial Attack Method for DRL-based Autonomous Driving Policies. (98%)
Junchao Fan; Xuyang Lei; Xiaolin Chang; Jelena Mišić; Vojislav B. Mišić
  Despite significant advancements in deep reinforcement learning (DRL)-based
autonomous driving policies, these policies still exhibit vulnerability to
adversarial attacks. This vulnerability poses a formidable challenge to the
practical deployment of these policies in autonomous driving. Designing
effective adversarial attacks is an indispensable prerequisite for enhancing
the robustness of these policies. In view of this, we present a novel stealthy
and efficient adversarial attack method for DRL-based autonomous driving
policies. Specifically, we introduce a DRL-based adversary designed to trigger
safety violations (e.g., collisions) by injecting adversarial samples at
critical moments. We model the attack as a mixed-integer optimization problem
and formulate it as a Markov decision process. Then, we train the adversary to
learn the optimal policy for attacking at critical moments without domain
knowledge. Furthermore, we introduce attack-related information and a
trajectory clipping method to enhance the learning capability of the adversary.
Finally, we validate our method in an unprotected left-turn scenario across
different traffic densities. The experimental results show that our method
achieves more than 90% collision rate within three attacks in most cases.
Furthermore, our method achieves more than 130% improvement in attack
efficiency compared to the unlimited attack method.


http://arxiv.org/abs/2412.04510
A Taxonomy of System-Level Attacks on Deep Learning Models in Autonomous Vehicles. (76%)
Masoud Jamshidiyan Tehrani; Jinhan Kim; Rosmael Zidane Lekeufack Foulefack; Alessandro Marchetto; Paolo Tonella
  The advent of deep learning and its astonishing performance in perception
tasks, such as object recognition and classification, has enabled its usage in
complex systems, including autonomous vehicles. On the other hand, deep
learning models are susceptible to mis-predictions when small, adversarial
changes are introduced into their input. Such mis-predictions can be triggered
in the real world and can propagate to a failure of the entire system, as
opposed to a localized mis-prediction. In recent years, a growing number of
research works have investigated ways to mount attacks against autonomous
vehicles that exploit deep learning components for perception tasks. Such
attacks are directed toward elements of the environment where these systems
operate and their effectiveness is assessed in terms of system-level failures
triggered by them. There has been however no systematic attempt to analyze and
categorize such attacks. In this paper, we present the first taxonomy of
system-level attacks against autonomous vehicles. We constructed our taxonomy
by first collecting 8,831 papers, then filtering them down to 1,125 candidates
and eventually selecting a set of 19 highly relevant papers that satisfy all
inclusion criteria. Then, we tagged them with taxonomy categories, involving
three assessors per paper. The resulting taxonomy includes 12 top-level
categories and several sub-categories. The taxonomy allowed us to investigate
the attack features, the most attacked components, the underlying threat
models, and the propagation chains from input perturbation to system-level
failure. We distilled several lessons for practitioners and identified possible
directions for future work for researchers.


http://arxiv.org/abs/2412.03441
PBP: Post-training Backdoor Purification for Malware Classifiers. (76%)
Dung Thuy Nguyen; Ngoc N. Tran; Taylor T. Johnson; Kevin Leach
  In recent years, the rise of machine learning (ML) in cybersecurity has
brought new challenges, including the increasing threat of backdoor poisoning
attacks on ML malware classifiers. For instance, adversaries could inject
malicious samples into public malware repositories, contaminating the training
data and potentially misclassifying malware by the ML model. Current
countermeasures predominantly focus on detecting poisoned samples by leveraging
disagreements within the outputs of a diverse set of ensemble models on
training data points. However, these methods are not suitable for scenarios
where Machine Learning-as-a-Service (MLaaS) is used or when users aim to remove
backdoors from a model after it has been trained. Addressing this scenario, we
introduce PBP, a post-training defense for malware classifiers that mitigates
various types of backdoor embeddings without assuming any specific backdoor
embedding mechanism. Our method exploits the influence of backdoor attacks on
the activation distribution of neural networks, independent of the
trigger-embedding method. In the presence of a backdoor attack, the activation
distribution of each layer is distorted into a mixture of distributions. By
regulating the statistics of the batch normalization layers, we can guide a
backdoored model to perform similarly to a clean one. Our method demonstrates
substantial advantages over several state-of-the-art methods, as evidenced by
experiments on two datasets, two types of backdoor methods, and various attack
configurations. Notably, our approach requires only a small portion of the
training data -- only 1\% -- to purify the backdoor and reduce the attack
success rate from 100\% to almost 0\%, a 100-fold improvement over the baseline
methods. Our code is available at
\url{https://github.com/judydnguyen/pbp-backdoor-purification-official}.


http://arxiv.org/abs/2412.03154
Testing Neural Network Verifiers: A Soundness Benchmark with Hidden Counterexamples. (13%)
Xingjian Zhou; Hongji Xu; Andy Xu; Zhouxing Shi; Cho-Jui Hsieh; Huan Zhang
  In recent years, many neural network (NN) verifiers have been developed to
formally verify certain properties of neural networks such as robustness.
Although many benchmarks have been constructed to evaluate the performance of
NN verifiers, they typically lack a ground-truth for hard instances where no
current verifier can verify and no counterexample can be found, which makes it
difficult to check the soundness of a new verifier if it claims to verify hard
instances which no other verifier can do. We propose to develop a soundness
benchmark for NN verification. Our benchmark contains instances with
deliberately inserted counterexamples while we also try to hide the
counterexamples from regular adversarial attacks which can be used for finding
counterexamples. We design a training method to produce neural networks with
such hidden counterexamples. Our benchmark aims to be used for testing the
soundness of NN verifiers and identifying falsely claimed verifiability when it
is known that hidden counterexamples exist. We systematically construct our
benchmark and generate instances across diverse model architectures, activation
functions, input sizes, and perturbation radii. We demonstrate that our
benchmark successfully identifies bugs in state-of-the-art NN verifiers, as
well as synthetic bugs, providing a crucial step toward enhancing the
reliability of testing NN verifiers. Our code is available at
https://github.com/MVP-Harry/SoundnessBench and our benchmark is available at
https://huggingface.co/datasets/SoundnessBench/SoundnessBench.


http://arxiv.org/abs/2412.03283
Black-Box Forgery Attacks on Semantic Watermarks for Diffusion Models. (12%)
Andreas Müller; Denis Lukovnikov; Jonas Thietke; Asja Fischer; Erwin Quiring
  Integrating watermarking into the generation process of latent diffusion
models (LDMs) simplifies detection and attribution of generated content.
Semantic watermarks, such as Tree-Rings and Gaussian Shading, represent a novel
class of watermarking techniques that are easy to implement and highly robust
against various perturbations. However, our work demonstrates a fundamental
security vulnerability of semantic watermarks. We show that attackers can
leverage unrelated models, even with different latent spaces and architectures
(UNet vs DiT), to perform powerful and realistic forgery attacks. Specifically,
we design two watermark forgery attacks. The first imprints a targeted
watermark into real images by manipulating the latent representation of an
arbitrary image in an unrelated LDM to get closer to the latent representation
of a watermarked image. We also show that this technique can be used for
watermark removal. The second attack generates new images with the target
watermark by inverting a watermarked image and re-generating it with an
arbitrary prompt. Both attacks just need a single reference image with the
target watermark. Overall, our findings question the applicability of semantic
watermarks by revealing that attackers can easily forge or remove these
watermarks under realistic conditions.


http://arxiv.org/abs/2412.03682
Designing DNNs for a trade-off between robustness and processing performance in embedded devices. (11%)
Jon Gutiérrez-Zaballa; Koldo Basterretxea; Javier Echanobe
  Machine learning-based embedded systems employed in safety-critical
applications such as aerospace and autonomous driving need to be robust against
perturbations produced by soft errors. Soft errors are an increasing concern in
modern digital processors since smaller transistor geometries and lower
voltages give electronic devices a higher sensitivity to background radiation.
The resilience of deep neural network (DNN) models to perturbations in their
parameters is determined, to a large extent, by the structure of the model
itself, and also by the selected numerical representation and used arithmetic
precision. When compression techniques such as model pruning and model
quantization are applied to reduce memory footprint and computational
complexity for deployment, both model structure and numerical representation
are modified and thus, soft error robustness also changes. In this sense,
although the choice of activation functions (AFs) in DNN models is frequently
ignored, it conditions not only their accuracy and trainability, but also
compressibility rates and numerical robustness. This paper investigates the
suitability of using bounded AFs to improve model robustness against DNN
parameter perturbations, assessing at the same time the impact of this choice
on deployment in terms of model accuracy, compressibility, and computational
burden. In particular, we analyze encoder-decoder fully convolutional models
aimed at performing semantic segmentation tasks on hyperspectral images for
scene understanding in autonomous driving. Deployment characterization is
performed experimentally on an AMD-Xilinx's KV260 SoM.


http://arxiv.org/abs/2412.03453
Pre-trained Multiple Latent Variable Generative Models are good defenders against Adversarial Attacks. (4%)
Dario Serez; Marco Cristani; Bue Alessio Del; Vittorio Murino; Pietro Morerio
  Attackers can deliberately perturb classifiers' input with subtle noise,
altering final predictions. Among proposed countermeasures, adversarial
purification employs generative networks to preprocess input images, filtering
out adversarial noise. In this study, we propose specific generators, defined
Multiple Latent Variable Generative Models (MLVGMs), for adversarial
purification. These models possess multiple latent variables that naturally
disentangle coarse from fine features. Taking advantage of these properties, we
autoencode images to maintain class-relevant information, while discarding and
re-sampling any detail, including adversarial noise. The procedure is
completely training-free, exploring the generalization abilities of pre-trained
MLVGMs on the adversarial purification downstream task. Despite the lack of
large models, trained on billions of samples, we show that smaller MLVGMs are
already competitive with traditional methods, and can be used as foundation
models. Official code released at https://github.com/SerezD/gen_adversarial.


http://arxiv.org/abs/2412.03630
Evaluating Single Event Upsets in Deep Neural Networks for Semantic Segmentation: an embedded system perspective. (1%)
Jon Gutiérrez-Zaballa; Koldo Basterretxea; Javier Echanobe
  As the deployment of artifical intelligence (AI) algorithms at edge devices
becomes increasingly prevalent, enhancing the robustness and reliability of
autonomous AI-based perception and decision systems is becoming as relevant as
precision and performance, especially in applications areas considered
safety-critical such as autonomous driving and aerospace. This paper delves
into the robustness assessment in embedded Deep Neural Networks (DNNs),
particularly focusing on the impact of parameter perturbations produced by
single event upsets (SEUs) on convolutional neural networks (CNN) for image
semantic segmentation. By scrutinizing the layer-by-layer and bit-by-bit
sensitivity of various encoder-decoder models to soft errors, this study
thoroughly investigates the vulnerability of segmentation DNNs to SEUs and
evaluates the consequences of techniques like model pruning and parameter
quantization on the robustness of compressed models aimed at embedded
implementations. The findings offer valuable insights into the mechanisms
underlying SEU-induced failures that allow for evaluating the robustness of
DNNs once trained in advance. Moreover, based on the collected data, we propose
a set of practical lightweight error mitigation techniques with no memory or
computational cost suitable for resource-constrained deployments. The code used
to perform the fault injection (FI) campaign is available at
https://github.com/jonGuti13/TensorFI2 , while the code to implement proposed
techniques is available at https://github.com/jonGuti13/parameterProtection .


http://arxiv.org/abs/2412.02270
Sustainable Self-evolution Adversarial Training. (99%)
Wenxuan Wang; Chenglei Wang; Huihui Qi; Menghao Ye; Xuelin Qian; Peng Wang; Yanning Zhang
  With the wide application of deep neural network models in various computer
vision tasks, there has been a proliferation of adversarial example generation
strategies aimed at deeply exploring model security. However, existing
adversarial training defense models, which rely on single or limited types of
attacks under a one-time learning process, struggle to adapt to the dynamic and
evolving nature of attack methods. Therefore, to achieve defense performance
improvements for models in long-term applications, we propose a novel
Sustainable Self-Evolution Adversarial Training (SSEAT) framework.
Specifically, we introduce a continual adversarial defense pipeline to realize
learning from various kinds of adversarial examples across multiple stages.
Additionally, to address the issue of model catastrophic forgetting caused by
continual learning from ongoing novel attacks, we propose an adversarial data
replay module to better select more diverse and key relearning data.
Furthermore, we design a consistency regularization strategy to encourage
current defense models to learn more from previously trained ones, guiding them
to retain more past knowledge and maintain accuracy on clean samples. Extensive
experiments have been conducted to verify the efficacy of the proposed SSEAT
defense method, which demonstrates superior defense performance and
classification accuracy compared to competitors.


http://arxiv.org/abs/2412.02803
Gaussian Splatting Under Attack: Investigating Adversarial Noise in 3D Objects. (99%)
Abdurrahman Zeybey; Mehmet Ergezer; Tommy Nguyen
  3D Gaussian Splatting has advanced radiance field reconstruction, enabling
high-quality view synthesis and fast rendering in 3D modeling. While
adversarial attacks on object detection models are well-studied for 2D images,
their impact on 3D models remains underexplored. This work introduces the
Masked Iterative Fast Gradient Sign Method (M-IFGSM), designed to generate
adversarial noise targeting the CLIP vision-language model. M-IFGSM
specifically alters the object of interest by focusing perturbations on masked
regions, degrading the performance of CLIP's zero-shot object detection
capability when applied to 3D models. Using eight objects from the Common
Objects 3D (CO3D) dataset, we demonstrate that our method effectively reduces
the accuracy and confidence of the model, with adversarial noise being nearly
imperceptible to human observers. The top-1 accuracy in original model renders
drops from 95.4\% to 12.5\% for train images and from 91.2\% to 35.4\% for test
images, with confidence levels reflecting this shift from true classification
to misclassification, underscoring the risks of adversarial attacks on 3D
models in applications such as autonomous driving, robotics, and surveillance.
The significance of this research lies in its potential to expose
vulnerabilities in modern 3D vision models, including radiance fields,
prompting the development of more robust defenses and security measures in
critical real-world applications.


http://arxiv.org/abs/2412.02343
Multi-Granularity Tibetan Textual Adversarial Attack Method Based on Masked Language Model. (98%)
Xi Cao; Nuo Qun; Quzong Gesang; Yulei Zhu; Trashi Nyima
  In social media, neural network models have been applied to hate speech
detection, sentiment analysis, etc., but neural network models are susceptible
to adversarial attacks. For instance, in a text classification task, the
attacker elaborately introduces perturbations to the original texts that hardly
alter the original semantics in order to trick the model into making different
predictions. By studying textual adversarial attack methods, the robustness of
language models can be evaluated and then improved. Currently, most of the
research in this field focuses on English, and there is also a certain amount
of research on Chinese. However, there is little research targeting Chinese
minority languages. With the rapid development of artificial intelligence
technology and the emergence of Chinese minority language models, textual
adversarial attacks become a new challenge for the information processing of
Chinese minority languages. In response to this situation, we propose a
multi-granularity Tibetan textual adversarial attack method based on masked
language models called TSTricker. We utilize the masked language models to
generate candidate substitution syllables or words, adopt the scoring mechanism
to determine the substitution order, and then conduct the attack method on
several fine-tuned victim models. The experimental results show that TSTricker
reduces the accuracy of the classification models by more than 28.70% and makes
the classification models change the predictions of more than 90.60% of the
samples, which has an evidently higher attack effect than the baseline method.


http://arxiv.org/abs/2412.02323
Pay Attention to the Robustness of Chinese Minority Language Models! Syllable-level Textual Adversarial Attack on Tibetan Script. (98%)
Xi Cao; Dolma Dawa; Nuo Qun; Trashi Nyima
  The textual adversarial attack refers to an attack method in which the
attacker adds imperceptible perturbations to the original texts by elaborate
design so that the NLP (natural language processing) model produces false
judgments. This method is also used to evaluate the robustness of NLP models.
Currently, most of the research in this field focuses on English, and there is
also a certain amount of research on Chinese. However, to the best of our
knowledge, there is little research targeting Chinese minority languages.
Textual adversarial attacks are a new challenge for the information processing
of Chinese minority languages. In response to this situation, we propose a
Tibetan syllable-level black-box textual adversarial attack called TSAttacker
based on syllable cosine distance and scoring mechanism. And then, we conduct
TSAttacker on six models generated by fine-tuning two PLMs (pre-trained
language models) for three downstream tasks. The experiment results show that
TSAttacker is effective and generates high-quality adversarial samples. In
addition, the robustness of the involved models still has much room for
improvement.


http://arxiv.org/abs/2412.02171
Underload: Defending against Latency Attacks for Object Detectors on Edge Devices. (93%)
Tianyi Zhejiang University, Hangzhou, China Wang; Zichen Zhejiang University, Hangzhou, China Wang; Cong Zhejiang University, Hangzhou, China Wang; Yuanchao Zhejiang University, Hangzhou, China Shu; Ruilong Zhejiang University, Hangzhou, China Deng; Peng Zhejiang University, Hangzhou, China Cheng; Jiming Zhejiang University, Hangzhou, China Chen
  Object detection is a fundamental enabler for many real-time downstream
applications such as autonomous driving, augmented reality and supply chain
management. However, the algorithmic backbone of neural networks is brittle to
imperceptible perturbations in the system inputs, which were generally known as
misclassifying attacks. By targeting the real-time processing capability, a new
class of latency attacks are reported recently. They exploit new attack
surfaces in object detectors by creating a computational bottleneck in the
post-processing module, that leads to cascading failure and puts the real-time
downstream tasks at risks. In this work, we take an initial attempt to defend
against this attack via background-attentive adversarial training that is also
cognizant of the underlying hardware capabilities. We first draw system-level
connections between latency attack and hardware capacity across heterogeneous
GPU devices. Based on the particular adversarial behaviors, we utilize
objectness loss as a proxy and build background attention into the adversarial
training pipeline, and achieve a reasonable balance between clean and robust
accuracy. The extensive experiments demonstrate the defense effectiveness of
restoring real-time processing capability from $13$ FPS to $43$ FPS on Jetson
Orin NX, with a better trade-off between the clean and robust accuracy.


http://arxiv.org/abs/2412.02795
Hijacking Vision-and-Language Navigation Agents with Adversarial Environmental Attacks. (80%)
Zijiao Yang; Xiangxi Shi; Eric Slyman; Stefan Lee
  Assistive embodied agents that can be instructed in natural language to
perform tasks in open-world environments have the potential to significantly
impact labor tasks like manufacturing or in-home care -- benefiting the lives
of those who come to depend on them. In this work, we consider how this benefit
might be hijacked by local modifications in the appearance of the agent's
operating environment. Specifically, we take the popular Vision-and-Language
Navigation (VLN) task as a representative setting and develop a whitebox
adversarial attack that optimizes a 3D attack object's appearance to induce
desired behaviors in pretrained VLN agents that observe it in the environment.
We demonstrate that the proposed attack can cause VLN agents to ignore their
instructions and execute alternative actions after encountering the attack
object -- even for instructions and agent paths not considered when optimizing
the attack. For these novel settings, we find our attacks can induce
early-termination behaviors or divert an agent along an attacker-defined
multi-step trajectory. Under both conditions, environmental attacks
significantly reduce agent capabilities to successfully follow user
instructions.


http://arxiv.org/abs/2412.02371
TSCheater: Generating High-Quality Tibetan Adversarial Texts via Visual Similarity. (76%)
Xi Cao; Quzong Gesang; Yuan Sun; Nuo Qun; Tashi Nyima
  Language models based on deep neural networks are vulnerable to textual
adversarial attacks. While rich-resource languages like English are receiving
focused attention, Tibetan, a cross-border language, is gradually being studied
due to its abundant ancient literature and critical language strategy.
Currently, there are several Tibetan adversarial text generation methods, but
they do not fully consider the textual features of Tibetan script and
overestimate the quality of generated adversarial texts. To address this issue,
we propose a novel Tibetan adversarial text generation method called TSCheater,
which considers the characteristic of Tibetan encoding and the feature that
visually similar syllables have similar semantics. This method can also be
transferred to other abugidas, such as Devanagari script. We utilize a
self-constructed Tibetan syllable visual similarity database called TSVSDB to
generate substitution candidates and adopt a greedy algorithm-based scoring
mechanism to determine substitution order. After that, we conduct the method on
eight victim language models. Experimentally, TSCheater outperforms existing
methods in attack effectiveness, perturbation magnitude, semantic similarity,
visual similarity, and human acceptance. Finally, we construct the first
Tibetan adversarial robustness evaluation benchmark called AdvTS, which is
generated by existing methods and proofread by humans.


http://arxiv.org/abs/2412.02576
The Efficacy of Transfer-based No-box Attacks on Image Watermarking: A Pragmatic Analysis. (61%)
Qilong Wu; Varun Chandrasekaran
  Watermarking approaches are widely used to identify if images being
circulated are authentic or AI-generated. Determining the robustness of image
watermarking methods in the ``no-box'' setting, where the attacker is assumed
to have no knowledge about the watermarking model, is an interesting problem.
Our main finding is that evading the no-box setting is challenging: the success
of optimization-based transfer attacks (involving training surrogate models)
proposed in prior work~\cite{hu2024transfer} depends on impractical
assumptions, including (i) aligning the architecture and training
configurations of both the victim and attacker's surrogate watermarking models,
as well as (ii) a large number of surrogate models with potentially large
computational requirements. Relaxing these assumptions i.e., moving to a more
pragmatic threat model results in a failed attack, with an evasion rate at most
$21.1\%$. We show that when the configuration is mostly aligned, a simple
non-optimization attack we propose, OFT, with one single surrogate model can
already exceed the success of optimization-based efforts. Under the same
$\ell_\infty$ norm perturbation budget of $0.25$, prior
work~\citet{hu2024transfer} is comparable to or worse than OFT in $11$ out of
$12$ configurations and has a limited advantage on the remaining one. The code
used for all our experiments is available at
\url{https://github.com/Ardor-Wu/transfer}.


http://arxiv.org/abs/2412.02535
Defending Against Diverse Attacks in Federated Learning Through Consensus-Based Bi-Level Optimization. (22%)
Nicolás García Trillos; Aditya Kumar Akash; Sixu Li; Konstantin Riedl; Yuhua Zhu
  Adversarial attacks pose significant challenges in many machine learning
applications, particularly in the setting of distributed training and federated
learning, where malicious agents seek to corrupt the training process with the
goal of jeopardizing and compromising the performance and reliability of the
final models. In this paper, we address the problem of robust federated
learning in the presence of such attacks by formulating the training task as a
bi-level optimization problem. We conduct a theoretical analysis of the
resilience of consensus-based bi-level optimization (CB$^2$O), an interacting
multi-particle metaheuristic optimization method, in adversarial settings.
Specifically, we provide a global convergence analysis of CB$^2$O in mean-field
law in the presence of malicious agents, demonstrating the robustness of
CB$^2$O against a diverse range of attacks. Thereby, we offer insights into how
specific hyperparameter choices enable to mitigate adversarial effects. On the
practical side, we extend CB$^2$O to the clustered federated learning setting
by proposing FedCB$^2$O, a novel interacting multi-particle system, and design
a practical algorithm that addresses the demands of real-world applications.
Extensive experiments demonstrate the robustness of the FedCB$^2$O algorithm
against label-flipping attacks in decentralized clustered federated learning
scenarios, showcasing its effectiveness in practical contexts.


http://arxiv.org/abs/2412.02479
OODFace: Benchmarking Robustness of Face Recognition under Common Corruptions and Appearance Variations. (11%)
Caixin Kang; Yubo Chen; Shouwei Ruan; Shiji Zhao; Ruochen Zhang; Jiayi Wang; Shan Fu; Xingxing Wei
  With the rise of deep learning, facial recognition technology has seen
extensive research and rapid development. Although facial recognition is
considered a mature technology, we find that existing open-source models and
commercial algorithms lack robustness in certain real-world Out-of-Distribution
(OOD) scenarios, raising concerns about the reliability of these systems. In
this paper, we introduce OODFace, which explores the OOD challenges faced by
facial recognition models from two perspectives: common corruptions and
appearance variations. We systematically design 30 OOD scenarios across 9 major
categories tailored for facial recognition. By simulating these challenges on
public datasets, we establish three robustness benchmarks: LFW-C/V, CFP-FP-C/V,
and YTF-C/V. We then conduct extensive experiments on 19 different facial
recognition models and 3 commercial APIs, along with extended experiments on
face masks, Vision-Language Models (VLMs), and defense strategies to assess
their robustness. Based on the results, we draw several key insights,
highlighting the vulnerability of facial recognition systems to OOD data and
suggesting possible solutions. Additionally, we offer a unified toolkit that
includes all corruption and variation types, easily extendable to other
datasets. We hope that our benchmarks and findings can provide guidance for
future improvements in facial recognition model robustness.


http://arxiv.org/abs/2412.03002
AdvDreamer Unveils: Are Vision-Language Models Truly Ready for Real-World 3D Variations? (3%)
Shouwei Ruan; Hanqing Liu; Yao Huang; Xiaoqi Wang; Caixin Kang; Hang Su; Yinpeng Dong; Xingxing Wei
  Vision Language Models (VLMs) have exhibited remarkable generalization
capabilities, yet their robustness in dynamic real-world scenarios remains
largely unexplored. To systematically evaluate VLMs' robustness to real-world
3D variations, we propose AdvDreamer, the first framework that generates
physically reproducible adversarial 3D transformation (Adv-3DT) samples from
single-view images. AdvDreamer integrates advanced generative techniques with
two key innovations and aims to characterize the worst-case distributions of 3D
variations from natural images. To ensure adversarial effectiveness and method
generality, we introduce an Inverse Semantic Probability Objective that
executes adversarial optimization on fundamental vision-text alignment spaces,
which can be generalizable across different VLM architectures and downstream
tasks. To mitigate the distribution discrepancy between generated and
real-world samples while maintaining physical reproducibility, we design a
Naturalness Reward Model that provides regularization feedback during
adversarial optimization, preventing convergence towards hallucinated and
unnatural elements. Leveraging AdvDreamer, we establish MM3DTBench, the first
VQA dataset for benchmarking VLMs' 3D variations robustness. Extensive
evaluations on representative VLMs with diverse architectures highlight that 3D
variations in the real world may pose severe threats to model performance
across various tasks.


http://arxiv.org/abs/2412.02454
Gracefully Filtering Backdoor Samples for Generative Large Language Models without Retraining. (2%)
Zongru Wu; Pengzhou Cheng; Lingyong Fang; Zhuosheng Zhang; Gongshen Liu
  Backdoor attacks remain significant security threats to generative large
language models (LLMs). Since generative LLMs output sequences of
high-dimensional token logits instead of low-dimensional classification logits,
most existing backdoor defense methods designed for discriminative models like
BERT are ineffective for generative LLMs. Inspired by the observed differences
in learning behavior between backdoor and clean mapping in the frequency space,
we transform gradients of each training sample, directly influencing parameter
updates, into the frequency space. Our findings reveal a distinct separation
between the gradients of backdoor and clean samples in the frequency space.
Based on this phenomenon, we propose Gradient Clustering in the Frequency Space
for Backdoor Sample Filtering (GraCeFul), which leverages sample-wise gradients
in the frequency space to effectively identify backdoor samples without
requiring retraining LLMs. Experimental results show that GraCeFul outperforms
baselines significantly. Notably, GraCeFul exhibits remarkable computational
efficiency, achieving nearly 100% recall and F1 scores in identifying backdoor
samples, reducing the average success rate of various backdoor attacks to 0%
with negligible drops in clean accuracy across multiple free-style question
answering datasets. Additionally, GraCeFul generalizes to Llama-2 and Vicuna.
The codes are publicly available at https://github.com/ZrW00/GraceFul.


http://arxiv.org/abs/2412.02366
GenMix: Effective Data Augmentation with Generative Diffusion Model Image Editing. (1%)
Khawar Islam; Muhammad Zaigham Zaheer; Arif Mahmood; Karthik Nandakumar; Naveed Akhtar
  Data augmentation is widely used to enhance generalization in visual
classification tasks. However, traditional methods struggle when source and
target domains differ, as in domain adaptation, due to their inability to
address domain gaps. This paper introduces GenMix, a generalizable
prompt-guided generative data augmentation approach that enhances both
in-domain and cross-domain image classification. Our technique leverages image
editing to generate augmented images based on custom conditional prompts,
designed specifically for each problem type. By blending portions of the input
image with its edited generative counterpart and incorporating fractal
patterns, our approach mitigates unrealistic images and label ambiguity,
improving the performance and adversarial robustness of the resulting models.
Efficacy of our method is established with extensive experiments on eight
public datasets for general and fine-grained classification, in both in-domain
and cross-domain settings. Additionally, we demonstrate performance
improvements for self-supervised learning, learning with data scarcity, and
adversarial robustness. As compared to the existing state-of-the-art methods,
our technique achieves stronger performance across the board.


http://arxiv.org/abs/2412.01527
Traversing the Subspace of Adversarial Patches. (83%)
Jens Bayer; Stefan Becker; David Münch; Michael Arens; Jürgen Beyerer
  Despite ongoing research on the topic of adversarial examples in deep
learning for computer vision, some fundamentals of the nature of these attacks
remain unclear. As the manifold hypothesis posits, high-dimensional data tends
to be part of a low-dimensional manifold. To verify the thesis with adversarial
patches, this paper provides an analysis of a set of adversarial patches and
investigates the reconstruction abilities of three different dimensionality
reduction methods. Quantitatively, the performance of reconstructed patches in
an attack setting is measured and the impact of sampled patches from the latent
space during adversarial training is investigated. The evaluation is performed
on two publicly available datasets for person detection. The results indicate
that more sophisticated dimensionality reduction methods offer no advantages
over a simple principal component analysis.


http://arxiv.org/abs/2412.01440
DiffPatch: Generating Customizable Adversarial Patches using Diffusion Model. (82%)
Zhixiang Wang; Guangnan Ye; Xiaosen Wang; Siheng Chen; Zhibo Wang; Xingjun Ma; Yu-Gang Jiang
  Physical adversarial patches printed on clothing can easily allow individuals
to evade person detectors. However, most existing adversarial patch generation
methods prioritize attack effectiveness over stealthiness, resulting in patches
that are aesthetically unpleasing. Although existing methods using generative
adversarial networks or diffusion models can produce more natural-looking
patches, they often struggle to balance stealthiness with attack effectiveness
and lack flexibility for user customization. To address these challenges, we
propose a novel diffusion-based customizable patch generation framework termed
DiffPatch, specifically tailored for creating naturalistic and customizable
adversarial patches. Our approach enables users to utilize a reference image as
the source, rather than starting from random noise, and incorporates masks to
craft naturalistic patches of various shapes, not limited to squares. To
prevent the original semantics from being lost during the diffusion process, we
employ Null-text inversion to map random noise samples to a single input image
and generate patches through Incomplete Diffusion Optimization (IDO). Notably,
while maintaining a natural appearance, our method achieves a comparable attack
performance to state-of-the-art non-naturalistic patches when using similarly
sized attacks. Using DiffPatch, we have created a physical adversarial T-shirt
dataset, AdvPatch-1K, specifically targeting YOLOv5s. This dataset includes
over a thousand images across diverse scenarios, validating the effectiveness
of our attack in real-world environments. Moreover, it provides a valuable
resource for future research.


http://arxiv.org/abs/2412.01363
Exploring the Robustness of AI-Driven Tools in Digital Forensics: A Preliminary Study. (74%)
Silvia Lucia Sanna; Leonardo Regano; Davide Maiorca; Giorgio Giacinto
  Nowadays, many tools are used to facilitate forensic tasks about data
extraction and data analysis. In particular, some tools leverage Artificial
Intelligence (AI) to automatically label examined data into specific categories
(\ie, drugs, weapons, nudity). However, this raises a serious concern about the
robustness of the employed AI algorithms against adversarial attacks. Indeed,
some people may need to hide specific data to AI-based digital forensics tools,
thus manipulating the content so that the AI system does not recognize the
offensive/prohibited content and marks it at as suspicious to the analyst. This
could be seen as an anti-forensics attack scenario. For this reason, we
analyzed two of the most important forensics tools employing AI for data
classification: Magnet AI, used by Magnet Axiom, and Excire Photo AI, used by
X-Ways Forensics. We made preliminary tests using about $200$ images, other
$100$ sent in $3$ chats about pornography and teenage nudity, drugs and weapons
to understand how the tools label them. Moreover, we loaded some deepfake
images (images generated by AI forging real ones) of some actors to understand
if they would be classified in the same category as the original images. From
our preliminary study, we saw that the AI algorithm is not robust enough, as we
expected since these topics are still open research problems. For example, some
sexual images were not categorized as nudity, and some deepfakes were
categorized as the same real person, while the human eye can see the clear
nudity image or catch the difference between the deepfakes. Building on these
results and other state-of-the-art works, we provide some suggestions for
improving how digital forensics analysis tool leverage AI and their robustness
against adversarial attacks or different scenarios than the trained one.


http://arxiv.org/abs/2412.01756
Adversarial Sample-Based Approach for Tighter Privacy Auditing in Final Model-Only Scenarios. (69%)
Sangyeon Yoon; Wonje Jeung; Albert No
  Auditing Differentially Private Stochastic Gradient Descent (DP-SGD) in the
final model setting is challenging and often results in empirical lower bounds
that are significantly looser than theoretical privacy guarantees. We introduce
a novel auditing method that achieves tighter empirical lower bounds without
additional assumptions by crafting worst-case adversarial samples through
loss-based input-space auditing. Our approach surpasses traditional
canary-based heuristics and is effective in both white-box and black-box
scenarios. Specifically, with a theoretical privacy budget of $\varepsilon =
10.0$, our method achieves empirical lower bounds of $6.68$ in white-box
settings and $4.51$ in black-box settings, compared to the baseline of $4.11$
for MNIST. Moreover, we demonstrate that significant privacy auditing results
can be achieved using in-distribution (ID) samples as canaries, obtaining an
empirical lower bound of $4.33$ where traditional methods produce near-zero
leakage detection. Our work offers a practical framework for reliable and
accurate privacy auditing in differentially private machine learning.


http://arxiv.org/abs/2412.01646
Robust and Transferable Backdoor Attacks Against Deep Image Compression With Selective Frequency Prior. (67%)
Yi Yu; Yufei Wang; Wenhan Yang; Lanqing Guo; Shijian Lu; Ling-Yu Duan; Yap-Peng Tan; Alex C. Kot
  Recent advancements in deep learning-based compression techniques have
surpassed traditional methods. However, deep neural networks remain vulnerable
to backdoor attacks, where pre-defined triggers induce malicious behaviors.
This paper introduces a novel frequency-based trigger injection model for
launching backdoor attacks with multiple triggers on learned image compression
models. Inspired by the widely used DCT in compression codecs, triggers are
embedded in the DCT domain. We design attack objectives tailored to diverse
scenarios, including: 1) degrading compression quality in terms of bit-rate and
reconstruction accuracy; 2) targeting task-driven measures like face
recognition and semantic segmentation. To improve training efficiency, we
propose a dynamic loss function that balances loss terms with fewer
hyper-parameters, optimizing attack objectives effectively. For advanced
scenarios, we evaluate the attack's resistance to defensive preprocessing and
propose a two-stage training schedule with robust frequency selection to
enhance resilience. To improve cross-model and cross-domain transferability for
downstream tasks, we adjust the classification boundary in the attack loss
during training. Experiments show that our trigger injection models, combined
with minor modifications to encoder parameters, successfully inject multiple
backdoors and their triggers into a single compression model, demonstrating
strong performance and versatility. (*Due to the notification of arXiv "The
Abstract field cannot be longer than 1,920 characters", the appeared Abstract
is shortened. For the full Abstract, please download the Article.)


http://arxiv.org/abs/2412.01495
Adversarial Attacks on Hyperbolic Networks. (26%)
Spengler Max van; Jan Zahálka; Pascal Mettes
  As hyperbolic deep learning grows in popularity, so does the need for
adversarial robustness in the context of such a non-Euclidean geometry. To this
end, this paper proposes hyperbolic alternatives to the commonly used FGM and
PGD adversarial attacks. Through interpretable synthetic benchmarks and
experiments on existing datasets, we show how the existing and newly proposed
attacks differ. Moreover, we investigate the differences in adversarial
robustness between Euclidean and fully hyperbolic networks. We find that these
networks suffer from different types of vulnerabilities and that the newly
proposed hyperbolic attacks cannot address these differences. Therefore, we
conclude that the shifts in adversarial robustness are due to the models
learning distinct patterns resulting from their different geometries.


http://arxiv.org/abs/2412.02156
Compromising the Intelligence of Modern DNNs: On the Effectiveness of Targeted RowPress. (13%)
Ranyang Zhou; Jacqueline T. Liu; Sabbir Ahmed; Shaahin Angizi; Adnan Siraj Rakin
  Recent advancements in side-channel attacks have revealed the vulnerability
of modern Deep Neural Networks (DNNs) to malicious adversarial weight attacks.
The well-studied RowHammer attack has effectively compromised DNN performance
by inducing precise and deterministic bit-flips in the main memory (e.g.,
DRAM). Similarly, RowPress has emerged as another effective strategy for
flipping targeted bits in DRAM. However, the impact of RowPress on deep
learning applications has yet to be explored in the existing literature,
leaving a fundamental research question unanswered: How does RowPress compare
to RowHammer in leveraging bit-flip attacks to compromise DNN performance? This
paper is the first to address this question and evaluate the impact of RowPress
on DNN applications. We conduct a comparative analysis utilizing a novel
DRAM-profile-aware attack designed to capture the distinct bit-flip patterns
caused by RowHammer and RowPress. Eleven widely-used DNN architectures trained
on different benchmark datasets deployed on a Samsung DRAM chip conclusively
demonstrate that they suffer from a drastically more rapid performance
degradation under the RowPress attack compared to RowHammer. The difference in
the underlying attack mechanism of RowHammer and RowPress also renders existing
RowHammer mitigation mechanisms ineffective under RowPress. As a result,
RowPress introduces a new vulnerability paradigm for DNN compute platforms and
unveils the urgent need for corresponding protective measures.


http://arxiv.org/abs/2412.01528
CopyrightShield: Spatial Similarity Guided Backdoor Defense against Copyright Infringement in Diffusion Models. (10%)
Zhixiang Guo; Siyuan Liang; Aishan Liu; Dacheng Tao
  The diffusion model has gained significant attention due to its remarkable
data generation ability in fields such as image synthesis. However, its strong
memorization and replication abilities with respect to the training data also
make it a prime target for copyright infringement attacks. This paper provides
an in-depth analysis of the spatial similarity of replication in diffusion
model and leverages this key characteristic to design a method for detecting
poisoning data. By employing a joint assessment of spatial-level and
feature-level information from the detected segments, we effectively identify
covertly dispersed poisoned samples. Building upon detected poisoning data, we
propose a novel defense method specifically targeting copyright infringement
attacks by introducing a protection constraint term into the loss function to
mitigate the impact of poisoning. Extensive experimental results demonstrate
that our approach achieves an average F1 score of 0.709 in detecting copyright
infringement backdoors, resulting in an average increase of 68.1% in
First-Attack Epoch (FAE) and an average decrease of 51.4% in Copyright
Infringement Rate (CIR) of the poisoned model, effectively defending against
copyright infringement. Additionally, we introduce the concept of copyright
feature inversion, which aids in determining copyright responsibility and
expands the application scenarios of defense strategies.


http://arxiv.org/abs/2412.01154
R.I.P.: A Simple Black-box Attack on Continual Test-time Adaptation. (5%)
Trung-Hieu Hoang; Duc Minh Vo; Minh N. Do
  Test-time adaptation (TTA) has emerged as a promising solution to tackle the
continual domain shift in machine learning by allowing model parameters to
change at test time, via self-supervised learning on unlabeled testing data. At
the same time, it unfortunately opens the door to unforeseen vulnerabilities
for degradation over time. Through a simple theoretical continual TTA model, we
successfully identify a risk in the sampling process of testing data that could
easily degrade the performance of a continual TTA model. We name this risk as
Reusing of Incorrect Prediction (RIP) that TTA attackers can employ or as a
result of the unintended query from general TTA users. The risk posed by RIP is
also highly realistic, as it does not require prior knowledge of model
parameters or modification of testing samples. This simple requirement makes
RIP as the first black-box TTA attack algorithm that stands out from existing
white-box attempts. We extensively benchmark the performance of the most recent
continual TTA approaches when facing the RIP attack, providing insights on its
success, and laying out potential roadmaps that could enhance the resilience of
future continual TTA systems.


http://arxiv.org/abs/2412.01975
Reactive Synthesis of Sensor Revealing Strategies in Hypergames on Graphs. (1%)
Sumukha Udupa; Ahmed Hemida; Charles A. Kamhoua; Jie Fu
  In many security applications of cyber-physical systems, a system designer
must guarantee that critical missions are satisfied against attacks in the
sensors and actuators of the CPS. Traditional security design of CPSs often
assume that attackers have complete knowledge of the system. In this article,
we introduce a class of deception techniques and study how to leverage
asymmetric information created by deception to strengthen CPS security.
Consider an adversarial interaction between a CPS defender and an attacker, who
can perform sensor jamming attacks. To mitigate such attacks, the defender
introduces asymmetrical information by deploying a "hidden sensor," whose
presence is initially undisclosed but can be revealed if queried. We introduce
hypergames on graphs to model this game with asymmetric information. Building
on the solution concept called subjective rationalizable strategies in
hypergames, we identify two stages in the game: An initial game stage where the
defender commits to a strategy perceived rationalizable by the attacker until
he deviates from the equilibrium in the attacker's perceptual game; Upon the
deviation, a delay-attack game stage starts where the defender plays against
the attacker, who has a bounded delay in attacking the sensor being revealed.
Based on backward induction, we develop an algorithm that determines, for any
given state, if the defender can benefit from hiding a sensor and revealing it
later. If the answer is affirmative, the algorithm outputs a sensor revealing
strategy to determine when to reveal the sensor during dynamic interactions. We
demonstrate the effectiveness of our deceptive strategies through two case
studies related to CPS security applications.


http://arxiv.org/abs/2412.01127
Precision Profile Pollution Attack on Sequential Recommenders via Influence Function. (1%)
Xiaoyu Du; Yingying Chen; Yang Zhang; Jinhui Tang
  Sequential recommendation approaches have demonstrated remarkable proficiency
in modeling user preferences. Nevertheless, they are susceptible to profile
pollution attacks (PPA), wherein items are introduced into a user's interaction
history deliberately to influence the recommendation list. Since retraining the
model for each polluted item is time-consuming, recent PPAs estimate item
influence based on gradient directions to identify the most effective attack
candidates. However, the actual item representations diverge significantly from
the gradients, resulting in disparate outcomes.To tackle this challenge, we
introduce an INFluence Function-based Attack approach INFAttack that offers a
more accurate estimation of the influence of polluting items. Specifically, we
calculate the modifications to the original model using the influence function
when generating polluted sequences by introducing specific items. Subsequently,
we choose the sequence that has been most significantly influenced to
substitute the original sequence, thus promoting the target item. Comprehensive
experiments conducted on five real-world datasets illustrate that INFAttack
surpasses all baseline methods and consistently delivers stable attack
performance for both popular and unpopular items.


http://arxiv.org/abs/2412.00696
Intermediate Outputs Are More Sensitive Than You Think. (61%)
Tao Huang; Qingyu Huang; Jiayang Meng
  The increasing reliance on deep computer vision models that process sensitive
data has raised significant privacy concerns, particularly regarding the
exposure of intermediate results in hidden layers. While traditional privacy
risk assessment techniques focus on protecting overall model outputs, they
often overlook vulnerabilities within these intermediate representations.
Current privacy risk assessment techniques typically rely on specific attack
simulations to assess risk, which can be computationally expensive and
incomplete. This paper introduces a novel approach to measuring privacy risks
in deep computer vision models based on the Degrees of Freedom (DoF) and
sensitivity of intermediate outputs, without requiring adversarial attack
simulations. We propose a framework that leverages DoF to evaluate the amount
of information retained in each layer and combines this with the rank of the
Jacobian matrix to assess sensitivity to input variations. This dual analysis
enables systematic measurement of privacy risks at various model layers. Our
experimental validation on real-world datasets demonstrates the effectiveness
of this approach in providing deeper insights into privacy risks associated
with intermediate representations.


http://arxiv.org/abs/2412.01101
Hiding Faces in Plain Sight: Defending DeepFakes by Disrupting Face Detection. (16%)
Delong Zhu; Yuezun Li; Baoyuan Wu; Jiaran Zhou; Zhibo Wang; Siwei Lyu
  This paper investigates the feasibility of a proactive DeepFake defense
framework, {\em FacePosion}, to prevent individuals from becoming victims of
DeepFake videos by sabotaging face detection. The motivation stems from the
reliance of most DeepFake methods on face detectors to automatically extract
victim faces from videos for training or synthesis (testing). Once the face
detectors malfunction, the extracted faces will be distorted or incorrect,
subsequently disrupting the training or synthesis of the DeepFake model. To
achieve this, we adapt various adversarial attacks with a dedicated design for
this purpose and thoroughly analyze their feasibility. Based on FacePoison, we
introduce {\em VideoFacePoison}, a strategy that propagates FacePoison across
video frames rather than applying them individually to each frame. This
strategy can largely reduce the computational overhead while retaining the
favorable attack performance. Our method is validated on five face detectors,
and extensive experiments against eleven different DeepFake models demonstrate
the effectiveness of disrupting face detectors to hinder DeepFake generation.


http://arxiv.org/abs/2412.00797
Online Poisoning Attack Against Reinforcement Learning under Black-box Environments. (11%)
Jianhui Li; Bokang Zhang; Junfeng Wu
  This paper proposes an online environment poisoning algorithm tailored for
reinforcement learning agents operating in a black-box setting, where an
adversary deliberately manipulates training data to lead the agent toward a
mischievous policy. In contrast to prior studies that primarily investigate
white-box settings, we focus on a scenario characterized by \textit{unknown}
environment dynamics to the attacker and a \textit{flexible} reinforcement
learning algorithm employed by the targeted agent. We first propose an attack
scheme that is capable of poisoning the reward functions and state transitions.
The poisoning task is formalized as a constrained optimization problem,
following the framework of \cite{ma2019policy}. Given the transition
probabilities are unknown to the attacker in a black-box environment, we apply
a stochastic gradient descent algorithm, where the exact gradients are
approximated using sample-based estimates. A penalty-based method along with a
bilevel reformulation is then employed to transform the problem into an
unconstrained counterpart and to circumvent the double-sampling issue. The
algorithm's effectiveness is validated through a maze environment.


http://arxiv.org/abs/2412.00404
Hard-Label Black-Box Attacks on 3D Point Clouds. (99%)
Daizong Liu; Yunbo Tao; Pan Zhou; Wei Hu
  With the maturity of depth sensors in various 3D safety-critical
applications, 3D point cloud models have been shown to be vulnerable to
adversarial attacks. Almost all existing 3D attackers simply follow the
white-box or black-box setting to iteratively update coordinate perturbations
based on back-propagated or estimated gradients. However, these methods are
hard to deploy in real-world scenarios (no model details are provided) as they
severely rely on parameters or output logits of victim models. To this end, we
propose point cloud attacks from a more practical setting, i.e., hard-label
black-box attack, in which attackers can only access the prediction label of 3D
input. We introduce a novel 3D attack method based on a new spectrum-aware
decision boundary algorithm to generate high-quality adversarial samples. In
particular, we first construct a class-aware model decision boundary, by
developing a learnable spectrum-fusion strategy to adaptively fuse point clouds
of different classes in the spectral domain, aiming to craft their intermediate
samples without distorting the original geometry. Then, we devise an iterative
coordinate-spectrum optimization method with curvature-aware boundary search to
move the intermediate sample along the decision boundary for generating
adversarial point clouds with trivial perturbations. Experiments demonstrate
that our attack competitively outperforms existing white/black-box attackers in
terms of attack performance and adversary quality.


http://arxiv.org/abs/2412.00621
Exposing LLM Vulnerabilities: Adversarial Scam Detection and Performance. (69%)
Chen-Wei Chang; Shailik Sarkar; Shutonu Mitra; Qi Zhang; Hossein Salemi; Hemant Purohit; Fengxiu Zhang; Michin Hong; Jin-Hee Cho; Chang-Tien Lu
  Can we trust Large Language Models (LLMs) to accurately predict scam? This
paper investigates the vulnerabilities of LLMs when facing adversarial scam
messages for the task of scam detection. We addressed this issue by creating a
comprehensive dataset with fine-grained labels of scam messages, including both
original and adversarial scam messages. The dataset extended traditional binary
classes for the scam detection task into more nuanced scam types. Our analysis
showed how adversarial examples took advantage of vulnerabilities of a LLM,
leading to high misclassification rate. We evaluated the performance of LLMs on
these adversarial scam messages and proposed strategies to improve their
robustness.


http://arxiv.org/abs/2412.00537
Exact Certification of (Graph) Neural Networks Against Label Poisoning. (22%)
Mahalakshmi Sabanayagam; Lukas Gosch; Stephan Günnemann; Debarghya Ghoshdastidar
  Machine learning models are highly vulnerable to label flipping, i.e., the
adversarial modification (poisoning) of training labels to compromise
performance. Thus, deriving robustness certificates is important to guarantee
that test predictions remain unaffected and to understand worst-case robustness
behavior. However, for Graph Neural Networks (GNNs), the problem of certifying
label flipping has so far been unsolved. We change this by introducing an exact
certification method, deriving both sample-wise and collective certificates.
Our method leverages the Neural Tangent Kernel (NTK) to capture the training
dynamics of wide networks enabling us to reformulate the bilevel optimization
problem representing label flipping into a Mixed-Integer Linear Program (MILP).
We apply our method to certify a broad range of GNN architectures in node
classification tasks. Thereby, concerning the worst-case robustness to label
flipping: $(i)$ we establish hierarchies of GNNs on different benchmark graphs;
$(ii)$ quantify the effect of architectural choices such as activations, depth
and skip-connections; and surprisingly, $(iii)$ uncover a novel phenomenon of
the robustness plateauing for intermediate perturbation budgets across all
investigated datasets and architectures. While we focus on GNNs, our
certificates are applicable to sufficiently wide NNs in general through their
NTK. Thus, our work presents the first exact certificate to a poisoning attack
ever derived for neural networks, which could be of independent interest.


http://arxiv.org/abs/2412.00473
Jailbreak Large Vision-Language Models Through Multi-Modal Linkage. (12%)
Yu Wang; Xiaofei Zhou; Yichen Wang; Geyuan Zhang; Tianxing He
  With the significant advancement of Large Vision-Language Models (VLMs),
concerns about their potential misuse and abuse have grown rapidly. Previous
studies have highlighted VLMs' vulnerability to jailbreak attacks, where
carefully crafted inputs can lead the model to produce content that violates
ethical and legal standards. However, existing methods struggle against
state-of-the-art VLMs like GPT-4o, due to the over-exposure of harmful content
and lack of stealthy malicious guidance. In this work, we propose a novel
jailbreak attack framework: Multi-Modal Linkage (MML) Attack. Drawing
inspiration from cryptography, MML utilizes an encryption-decryption process
across text and image modalities to mitigate over-exposure of malicious
information. To align the model's output with malicious intent covertly, MML
employs a technique called "evil alignment", framing the attack within a video
game production scenario. Comprehensive experiments demonstrate MML's
effectiveness. Specifically, MML jailbreaks GPT-4o with attack success rates of
97.80% on SafeBench, 98.81% on MM-SafeBench and 99.07% on HADES-Dataset. Our
code is available at https://github.com/wangyu-ovo/MML


http://arxiv.org/abs/2411.19853
Towards Class-wise Robustness Analysis. (99%)
Tejaswini Medi; Julia Grabinski; Margret Keuper
  While being very successful in solving many downstream tasks, the application
of deep neural networks is limited in real-life scenarios because of their
susceptibility to domain shifts such as common corruptions, and adversarial
attacks. The existence of adversarial examples and data corruption
significantly reduces the performance of deep classification models.
Researchers have made strides in developing robust neural architectures to
bolster decisions of deep classifiers. However, most of these works rely on
effective adversarial training methods, and predominantly focus on overall
model robustness, disregarding class-wise differences in robustness, which are
critical. Exploiting weakly robust classes is a potential avenue for attackers
to fool the image recognition models. Therefore, this study investigates
class-to-class biases across adversarially trained robust classification models
to understand their latent space structures and analyze their strong and weak
class-wise properties. We further assess the robustness of classes against
common corruptions and adversarial attacks, recognizing that class
vulnerability extends beyond the number of correct classifications for a
specific class. We find that the number of false positives of classes as
specific target classes significantly impacts their vulnerability to attacks.
Through our analysis on the Class False Positive Score, we assess a fair
evaluation of how susceptible each class is to misclassification.


http://arxiv.org/abs/2411.19479
FLARE: Towards Universal Dataset Purification against Backdoor Attacks. (81%)
Linshan Hou; Wei Luo; Zhongyun Hua; Songhua Chen; Leo Yu Zhang; Yiming Li
  Deep neural networks (DNNs) are susceptible to backdoor attacks, where
adversaries poison datasets with adversary-specified triggers to implant hidden
backdoors, enabling malicious manipulation of model predictions. Dataset
purification serves as a proactive defense by removing malicious training
samples to prevent backdoor injection at its source. We first reveal that the
current advanced purification methods rely on a latent assumption that the
backdoor connections between triggers and target labels in backdoor attacks are
simpler to learn than the benign features. We demonstrate that this assumption,
however, does not always hold, especially in all-to-all (A2A) and untargeted
(UT) attacks. As a result, purification methods that analyze the separation
between the poisoned and benign samples in the input-output space or the final
hidden layer space are less effective. We observe that this separability is not
confined to a single layer but varies across different hidden layers. Motivated
by this understanding, we propose FLARE, a universal purification method to
counter various backdoor attacks. FLARE aggregates abnormal activations from
all hidden layers to construct representations for clustering. To enhance
separation, FLARE develops an adaptive subspace selection algorithm to isolate
the optimal space for dividing an entire dataset into two clusters. FLARE
assesses the stability of each cluster and identifies the cluster with higher
stability as poisoned. Extensive evaluations on benchmark datasets demonstrate
the effectiveness of FLARE against 22 representative backdoor attacks,
including all-to-one (A2O), all-to-all (A2A), and untargeted (UT) attacks, and
its robustness to adaptive attacks.


http://arxiv.org/abs/2412.00324
Robust Table Integration in Data Lakes. (56%)
Daomin Ji; Hui Luo; Zhifeng Bao; Shane Culpepper
  In this paper, we investigate the challenge of integrating tables from data
lakes, focusing on three core tasks: 1) pairwise integrability judgment, which
determines whether a tuple pair in a table is integrable, accounting for any
occurrences of semantic equivalence or typographical errors; 2) integrable set
discovery, which aims to identify all integrable sets in a table based on
pairwise integrability judgments established in the first task; 3) multi-tuple
conflict resolution, which resolves conflicts among multiple tuples during
integration. We train a binary classifier to address the task of pairwise
integrability judgment. Given the scarcity of labeled data, we propose a
self-supervised adversarial contrastive learning algorithm to perform
classification, which incorporates data augmentation methods and adversarial
examples to autonomously generate new training data. Upon the output of
pairwise integrability judgment, each integrable set is considered as a
community, a densely connected sub-graph where nodes and edges correspond to
tuples in the table and their pairwise integrability, respectively. We proceed
to investigate various community detection algorithms to address the integrable
set discovery objective. Moving forward to tackle multi-tuple conflict
resolution, we introduce an novel in-context learning methodology. This
approach capitalizes on the knowledge embedded within pretrained large language
models to effectively resolve conflicts that arise when integrating multiple
tuples. Notably, our method minimizes the need for annotated data. Since no
suitable test collections are available for our tasks, we develop our own
benchmarks using two real-word dataset repositories: Real and Join. We conduct
extensive experiments on these benchmarks to validate the robustness and
applicability of our methodologies in the context of integrating tables within
data lakes.


http://arxiv.org/abs/2411.19508
On the Adversarial Robustness of Instruction-Tuned Large Language Models for Code. (38%)
Md Imran Hossen; Xiali Hei
  The advent of instruction-tuned Large Language Models designed for coding
tasks (Code LLMs) has transformed software engineering practices. However,
their robustness against various input challenges remains a critical concern.
This study introduces DegradePrompter, a novel method designed to
systematically evaluate the robustness of instruction-tuned Code LLMs. We
assess the impact of diverse input challenges on the functionality and
correctness of generated code using rigorous metrics and established
benchmarks. Our comprehensive evaluation includes five state-of-the-art
open-source models and three production-grade closed-source models, revealing
varying degrees of robustness. Open-source models demonstrate an increased
susceptibility to input perturbations, resulting in declines in functional
correctness ranging from 12% to 34%. In contrast, commercial models demonstrate
relatively greater resilience, with performance degradation ranging from 3% to
24%. To enhance the robustness of the models against these vulnerabilities, we
investigate a straightforward yet effective mitigation strategy. Our findings
highlight the need for robust defense mechanisms and comprehensive evaluations
during both the development and deployment phases to ensure the resilience and
reliability of automated code generation systems.


http://arxiv.org/abs/2411.19841
Parallel Stacked Aggregated Network for Voice Authentication in IoT-Enabled Smart Devices. (10%)
Awais Khan; Ijaz Ul Haq; Khalid Mahmood Malik
  Voice authentication on IoT-enabled smart devices has gained prominence in
recent years due to increasing concerns over user privacy and security. The
current authentication systems are vulnerable to different voice-spoofing
attacks (e.g., replay, voice cloning, and audio deepfakes) that mimic
legitimate voices to deceive authentication systems and enable fraudulent
activities (e.g., impersonation, unauthorized access, financial fraud, etc.).
Existing solutions are often designed to tackle a single type of attack,
leading to compromised performance against unseen attacks. On the other hand,
existing unified voice anti-spoofing solutions, not designed specifically for
IoT, possess complex architectures and thus cannot be deployed on IoT-enabled
smart devices. Additionally, most of these unified solutions exhibit
significant performance issues, including higher equal error rates or lower
accuracy for specific attacks. To overcome these issues, we present the
parallel stacked aggregation network (PSA-Net), a lightweight framework
designed as an anti-spoofing defense system for voice-controlled smart IoT
devices. The PSA-Net processes raw audios directly and eliminates the need for
dataset-dependent handcrafted features or pre-computed spectrograms.
Furthermore, PSA-Net employs a split-transform-aggregate approach, which
involves the segmentation of utterances, the extraction of intrinsic
differentiable embeddings through convolutions, and the aggregation of them to
distinguish legitimate from spoofed audios. In contrast to existing deep
Resnet-oriented solutions, we incorporate cardinality as an additional
dimension in our network, which enhances the PSA-Net ability to generalize
across diverse attacks. The results show that the PSA-Net achieves more
consistent performance for different attacks that exist in current
anti-spoofing solutions.


http://arxiv.org/abs/2412.00341
Fusing Physics-Driven Strategies and Cross-Modal Adversarial Learning: Toward Multi-Domain Applications. (1%)
Hana Satou; Alan Mitkiy
  The convergence of cross-modal adversarial learning and physics-driven
methods represents a cutting-edge direction for tackling challenges in complex
multi-modal tasks and scientific computing. This review focuses on
systematically analyzing how these two approaches can be synergistically
integrated to enhance performance and robustness across diverse application
domains. By addressing key obstacles such as modality discrepancies, limited
data availability, and insufficient model robustness, this paper highlights the
role of physics-based optimization frameworks in facilitating efficient and
interpretable adversarial perturbation generation. The review also explores
significant advancements in cross-modal adversarial learning, including
applications in tasks such as image cross-modal retrieval (e.g., infrared and
RGB matching), scientific computing (e.g., solving partial differential
equations), and optimization under physical consistency constraints in vision
systems. By examining theoretical foundations and experimental outcomes, this
study demonstrates the potential of combining these approaches to handle
complex scenarios and improve the security of multi-modal systems. Finally, we
outline future directions, proposing a novel framework that unifies physical
principles with adversarial optimization, providing a pathway for researchers
to develop robust and adaptable cross-modal learning methods with both
theoretical and practical significance.


http://arxiv.org/abs/2411.19688
SURE-VQA: Systematic Understanding of Robustness Evaluation in Medical VQA Tasks. (1%)
Kim-Celine Kahl; Selen Erkan; Jeremias Traub; Carsten T. Lüth; Klaus Maier-Hein; Lena Maier-Hein; Paul F. Jaeger
  Vision-Language Models (VLMs) have great potential in medical tasks, like
Visual Question Answering (VQA), where they could act as interactive assistants
for both patients and clinicians. Yet their robustness to distribution shifts
on unseen data remains a critical concern for safe deployment. Evaluating such
robustness requires a controlled experimental setup that allows for systematic
insights into the model's behavior. However, we demonstrate that current setups
fail to offer sufficiently thorough evaluations, limiting their ability to
accurately assess model robustness. To address this gap, our work introduces a
novel framework, called SURE-VQA, centered around three key requirements to
overcome the current pitfalls and systematically analyze the robustness of
VLMs: 1) Since robustness on synthetic shifts does not necessarily translate to
real-world shifts, robustness should be measured on real-world shifts that are
inherent to the VQA data; 2) Traditional token-matching metrics often fail to
capture underlying semantics, necessitating the use of large language models
(LLMs) for more accurate semantic evaluation; 3) Model performance often lacks
interpretability due to missing sanity baselines, thus meaningful baselines
should be reported that allow assessing the multimodal impact on the VLM. To
demonstrate the relevance of this framework, we conduct a study on the
robustness of various fine-tuning methods across three medical datasets with
four different types of distribution shifts. Our study reveals several
important findings: 1) Sanity baselines that do not utilize image data can
perform surprisingly well; 2) We confirm LoRA as the best-performing PEFT
method; 3) No PEFT method consistently outperforms others in terms of
robustness to shifts. Code is provided at https://github.com/IML-DKFZ/sure-vqa.


http://arxiv.org/abs/2412.00114
SceneTAP: Scene-Coherent Typographic Adversarial Planner against Vision-Language Models in Real-World Environments. (84%)
Yue Cao; Yun Xing; Jie Zhang; Di Lin; Tianwei Zhang; Ivor Tsang; Yang Liu; Qing Guo
  Large vision-language models (LVLMs) have shown remarkable capabilities in
interpreting visual content. While existing works demonstrate these models'
vulnerability to deliberately placed adversarial texts, such texts are often
easily identifiable as anomalous. In this paper, we present the first approach
to generate scene-coherent typographic adversarial attacks that mislead
advanced LVLMs while maintaining visual naturalness through the capability of
the LLM-based agent. Our approach addresses three critical questions: what
adversarial text to generate, where to place it within the scene, and how to
integrate it seamlessly. We propose a training-free, multi-modal LLM-driven
scene-coherent typographic adversarial planning (SceneTAP) that employs a
three-stage process: scene understanding, adversarial planning, and seamless
integration. The SceneTAP utilizes chain-of-thought reasoning to comprehend the
scene, formulate effective adversarial text, strategically plan its placement,
and provide detailed instructions for natural integration within the image.
This is followed by a scene-coherent TextDiffuser that executes the attack
using a local diffusion mechanism. We extend our method to real-world scenarios
by printing and placing generated patches in physical environments,
demonstrating its practical implications. Extensive experiments show that our
scene-coherent adversarial text successfully misleads state-of-the-art LVLMs,
including ChatGPT-4o, even after capturing new images of physical setups. Our
evaluations demonstrate a significant increase in attack success rates while
maintaining visual naturalness and contextual appropriateness. This work
highlights vulnerabilities in current vision-language models to sophisticated,
scene-coherent adversarial attacks and provides insights into potential defense
mechanisms.


http://arxiv.org/abs/2411.19335
PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuning. (69%)
Shenghui Li; Edith C. -H. Ngai; Fanghua Ye; Thiemo Voigt
  Federated Parameter-Efficient Fine-Tuning (FedPEFT) has emerged as a
promising paradigm for privacy-preserving and efficient adaptation of
Pre-trained Language Models (PLMs) in Federated Learning (FL) settings. It
preserves data privacy by keeping the data decentralized and training the model
on local devices, ensuring that raw data never leaves the user's device.
Moreover, the integration of PEFT methods such as LoRA significantly reduces
the number of trainable parameters compared to fine-tuning the entire model,
thereby minimizing communication costs and computational overhead. Despite its
potential, the security implications of FedPEFT remain underexplored. This
paper introduces a novel security threat to FedPEFT, termed PEFT-as-an-Attack
(PaaA), which exposes how PEFT can be exploited as an attack vector to
circumvent PLMs' safety alignment and generate harmful content in response to
malicious prompts. Our evaluation of PaaA reveals that with less than 1% of the
model's parameters set as trainable, and a small subset of clients acting
maliciously, the attack achieves an approximate 80% attack success rate using
representative PEFT methods such as LoRA. To mitigate this threat, we further
investigate potential defense strategies, including Robust Aggregation Schemes
(RASs) and Post-PEFT Safety Alignment (PPSA). However, our empirical analysis
highlights the limitations of these defenses, i.e., even the most advanced
RASs, such as DnC and ClippedClustering, struggle to defend against PaaA in
scenarios with highly heterogeneous data distributions. Similarly, while PPSA
can reduce attack success rates to below 10%, it severely degrades the model's
accuracy on the target task. Our results underscore the urgent need for more
effective defense mechanisms that simultaneously ensure security and maintain
the performance of the FedPEFT paradigm.


http://arxiv.org/abs/2411.18956
Random Sampling for Diffusion-based Adversarial Purification. (26%)
Jiancheng Zhang; Peiran Dong; Yongyong Chen; Yin-Ping Zhao; Song Guo
  Denoising Diffusion Probabilistic Models (DDPMs) have gained great attention
in adversarial purification. Current diffusion-based works focus on designing
effective condition-guided mechanisms while ignoring a fundamental problem,
i.e., the original DDPM sampling is intended for stable generation, which may
not be the optimal solution for adversarial purification. Inspired by the
stability of the Denoising Diffusion Implicit Model (DDIM), we propose an
opposite sampling scheme called random sampling. In brief, random sampling will
sample from a random noisy space during each diffusion process, while DDPM and
DDIM sampling will continuously sample from the adjacent or original noisy
space. Thus, random sampling obtains more randomness and achieves stronger
robustness against adversarial attacks. Correspondingly, we also introduce a
novel mediator conditional guidance to guarantee the consistency of the
prediction under the purified image and clean image input. To expand awareness
of guided diffusion purification, we conduct a detailed evaluation with
different sampling methods and our random sampling achieves an impressive
improvement in multiple settings. Leveraging mediator-guided random sampling,
we also establish a baseline method named DiffAP, which significantly
outperforms state-of-the-art (SOTA) approaches in performance and defensive
stability. Remarkably, under strong attack, our DiffAP even achieves a more
than 20% robustness advantage with 10$\times$ sampling acceleration.


http://arxiv.org/abs/2412.04495
Artificial intelligence and cybersecurity in banking sector: opportunities and risks. (12%)
Ana Kovacevic; Sonja D. Radenkovic; Dragana Nikolic
  The rapid advancements in artificial intelligence (AI) have presented new
opportunities for enhancing efficiency and economic competitiveness across
various industries, espcially in banking. Machine learning (ML), as a subset of
artificial intelligence, enables systems to adapt and learn from vast datasets,
revolutionizing decision-making processes, fraud detection, and customer
service automation. However, these innovations also introduce new challenges,
particularly in the realm of cybersecurity. Adversarial attacks, such as data
poisoning and evasion attacks, represent critical threats to machine learning
models, exploiting vulnerabilities to manipulate outcomes or compromise
sensitive information. Furthermore, this study highlights the dual-use nature
of AI tools, which can be used by malicious users. To address these challenges,
the paper emphasizes the importance of developing machine learning models with
key characteristics such as security, trust, resilience and robustness. These
features are essential to mitigating risks and ensuring the secure deployment
of AI technologies in banking sectors, where the protection of financial data
is paramount. The findings underscore the urgent need for enhanced
cybersecurity frameworks and continuous improvements in defensive mechanisms.
By exploring both opportunities and risks, this paper aims to guide the
responsible integration of AI in the banking sector, paving the way for
innovation while safeguarding against emerging threats.


http://arxiv.org/abs/2411.19117
Understanding and Improving Training-Free AI-Generated Image Detections with Vision Foundation Models. (11%)
Chung-Ting Tsai; Ching-Yun Ko; I-Hsin Chung; Yu-Chiang Frank Wang; Pin-Yu Chen
  The rapid advancement of generative models has introduced serious risks,
including deepfake techniques for facial synthesis and editing. Traditional
approaches rely on training classifiers and enhancing generalizability through
various feature extraction techniques. Meanwhile, training-free detection
methods address issues like limited data and overfitting by directly leveraging
statistical properties from vision foundation models to distinguish between
real and fake images. The current leading training-free approach, RIGID,
utilizes DINOv2 sensitivity to perturbations in image space for detecting fake
images, with fake image embeddings exhibiting greater sensitivity than those of
real images. This observation prompts us to investigate how detection
performance varies across model backbones, perturbation types, and datasets.
Our experiments reveal that detection performance is closely linked to model
robustness, with self-supervised (SSL) models providing more reliable
representations. While Gaussian noise effectively detects general objects, it
performs worse on facial images, whereas Gaussian blur is more effective due to
potential frequency artifacts. To further improve detection, we introduce
Contrastive Blur, which enhances performance on facial images, and MINDER
(MINimum distance DetEctoR), which addresses noise type bias, balancing
performance across domains. Beyond performance gains, our work offers valuable
insights for both the generative and detection communities, contributing to a
deeper understanding of model robustness property utilized for deepfake
detection.


http://arxiv.org/abs/2411.19075
LADDER: Multi-objective Backdoor Attack via Evolutionary Algorithm. (2%)
Dazhuang Liu; Yanqi Qiao; Rui Wang; Kaitai Liang; Georgios Smaragdakis
  Current black-box backdoor attacks in convolutional neural networks formulate
attack objective(s) as single-objective optimization problems in single domain.
Designing triggers in single domain harms semantics and trigger robustness as
well as introduces visual and spectral anomaly. This work proposes a
multi-objective black-box backdoor attack in dual domains via evolutionary
algorithm (LADDER), the first instance of achieving multiple attack objectives
simultaneously by optimizing triggers without requiring prior knowledge about
victim model. In particular, we formulate LADDER as a multi-objective
optimization problem (MOP) and solve it via multi-objective evolutionary
algorithm (MOEA). MOEA maintains a population of triggers with trade-offs among
attack objectives and uses non-dominated sort to drive triggers toward optimal
solutions. We further apply preference-based selection to MOEA to exclude
impractical triggers. We state that LADDER investigates a new dual-domain
perspective for trigger stealthiness by minimizing the anomaly between clean
and poisoned samples in the spectral domain. Lastly, the robustness against
preprocessing operations is achieved by pushing triggers to low-frequency
regions. Extensive experiments comprehensively showcase that LADDER achieves
attack effectiveness of at least 99%, attack robustness with 90.23% (50.09%
higher than state-of-the-art attacks on average), superior natural stealthiness
(1.12x to 196.74x improvement) and excellent spectral stealthiness (8.45x
enhancement) as compared to current stealthy attacks by the average $l_2$-norm
across 5 public datasets.


http://arxiv.org/abs/2411.19027
Enhancing Neural Network Robustness Against Fault Injection Through Non-linear Weight Transformations. (2%)
Ninnart Fuengfusin; Hakaru Tamukoh
  Deploying deep neural networks (DNNs) in real-world environments poses
challenges due to faults that can manifest in physical hardware from radiation,
aging, and temperature fluctuations. To address this, previous works have
focused on protecting DNNs via activation range restriction using clipped ReLU
and finding the optimal clipping threshold. However, this work instead focuses
on constraining DNN weights by applying saturated activation functions (SAFs):
Tanh, Arctan, and others. SAFs prevent faults from causing DNN weights to
become excessively large, which can lead to model failure. These methods not
only enhance the robustness of DNNs against fault injections but also improve
DNN performance by a small margin. Before deployment, DNNs are trained with
weights constrained by SAFs. During deployment, the weights without applied SAF
are written to mediums with faults. When read, weights with faults are applied
with SAFs and are used for inference. We demonstrate our proposed method across
three datasets (CIFAR10, CIFAR100, ImageNet 2012) and across three datatypes
(32-bit floating point (FP32), 16-bit floating point, and 8-bit fixed point).
We show that our method enables FP32 ResNet18 with ImageNet 2012 to operate at
a bit-error rate of 0.00001 with minor accuracy loss, while without the
proposed method, the FP32 DNN only produces random guesses. Furthermore, to
accelerate the training process, we demonstrate that an ImageNet 2012
pre-trained ResNet18 can be adapted to SAF by training for a few epochs with a
slight improvement in Top-1 accuracy while still ensuring robustness against
fault injection.


http://arxiv.org/abs/2411.18275
Visual Adversarial Attack on Vision-Language Models for Autonomous Driving. (99%)
Tianyuan Zhang; Lu Wang; Xinwei Zhang; Yitong Zhang; Boyi Jia; Siyuan Liang; Shengshan Hu; Qiang Fu; Aishan Liu; Xianglong Liu
  Vision-language models (VLMs) have significantly advanced autonomous driving
(AD) by enhancing reasoning capabilities. However, these models remain highly
vulnerable to adversarial attacks. While existing research has primarily
focused on general VLM attacks, the development of attacks tailored to the
safety-critical AD context has been largely overlooked. In this paper, we take
the first step toward designing adversarial attacks specifically targeting VLMs
in AD, exposing the substantial risks these attacks pose within this critical
domain. We identify two unique challenges for effective adversarial attacks on
AD VLMs: the variability of textual instructions and the time-series nature of
visual scenarios. To this end, we propose ADvLM, the first visual adversarial
attack framework specifically designed for VLMs in AD. Our framework introduces
Semantic-Invariant Induction, which uses a large language model to create a
diverse prompt library of textual instructions with consistent semantic
content, guided by semantic entropy. Building on this, we introduce
Scenario-Associated Enhancement, an approach where attention mechanisms select
key frames and perspectives within driving scenarios to optimize adversarial
perturbations that generalize across the entire scenario. Extensive experiments
on several AD VLMs over multiple benchmarks show that ADvLM achieves
state-of-the-art attack effectiveness. Moreover, real-world attack studies
further validate its applicability and potential in practice.


http://arxiv.org/abs/2411.18776
Fall Leaf Adversarial Attack on Traffic Sign Classification. (99%)
Anthony Etim; Jakub Szefer
  Adversarial input image perturbation attacks have emerged as a significant
threat to machine learning algorithms, particularly in image classification
setting. These attacks involve subtle perturbations to input images that cause
neural networks to misclassify the input images, even though the images remain
easily recognizable to humans. One critical area where adversarial attacks have
been demonstrated is in automotive systems where traffic sign classification
and recognition is critical, and where misclassified images can cause
autonomous systems to take wrong actions. This work presents a new class of
adversarial attacks. Unlike existing work that has focused on adversarial
perturbations that leverage human-made artifacts to cause the perturbations,
such as adding stickers, paint, or shining flashlights at traffic signs, this
work leverages nature-made artifacts: tree leaves. By leveraging nature-made
artifacts, the new class of attacks has plausible deniability: a fall leaf
stuck to a street sign could come from a near-by tree, rather than be placed
there by an malicious human attacker. To evaluate the new class of the
adversarial input image perturbation attacks, this work analyses how fall
leaves can cause misclassification in street signs. The work evaluates various
leaves from different species of trees, and considers various parameters such
as size, color due to tree leaf type, and rotation. The work demonstrates high
success rate for misclassification. The work also explores the correlation
between successful attacks and how they affect the edge detection, which is
critical in many image classification algorithms.


http://arxiv.org/abs/2411.18688
Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment. (67%)
Soumya Suvra Ghosal; Souradip Chakraborty; Vaibhav Singh; Tianrui Guan; Mengdi Wang; Ahmad Beirami; Furong Huang; Alvaro Velasquez; Dinesh Manocha; Amrit Singh Bedi
  With the widespread deployment of Multimodal Large Language Models (MLLMs)
for visual-reasoning tasks, improving their safety has become crucial. Recent
research indicates that despite training-time safety alignment, these models
remain vulnerable to jailbreak attacks. In this work, we first highlight an
important safety gap to describe that alignment achieved solely through safety
training may be insufficient against jailbreak attacks. To address this
vulnerability, we propose Immune, an inference-time defense framework that
leverages a safe reward model through controlled decoding to defend against
jailbreak attacks. Additionally, we provide a mathematical characterization of
Immune, offering provable guarantees against jailbreaks. Extensive evaluations
on diverse jailbreak benchmarks using recent MLLMs reveal that Immune
effectively enhances model safety while preserving the model's original
capabilities. For instance, against text-based jailbreak attacks on LLaVA-1.6,
Immune reduces the attack success rate by 57.82% and 16.78% compared to the
base MLLM and state-of-the-art defense strategy, respectively.


http://arxiv.org/abs/2411.18280
Neutralizing Backdoors through Information Conflicts for Large Language Models. (26%)
Chen Chen; Yuchen Sun; Xueluan Gong; Jiaxin Gao; Kwok-Yan Lam
  Large language models (LLMs) have seen significant advancements, achieving
superior performance in various Natural Language Processing (NLP) tasks, from
understanding to reasoning. However, they remain vulnerable to backdoor
attacks, where models behave normally for standard queries but generate harmful
responses or unintended output when specific triggers are activated. Existing
backdoor defenses often suffer from drawbacks that they either focus on
detection without removal, rely on rigid assumptions about trigger properties,
or prove to be ineffective against advanced attacks like multi-trigger
backdoors. In this paper, we present a novel method to eliminate backdoor
behaviors from LLMs through the construction of information conflicts using
both internal and external mechanisms. Internally, we leverage a lightweight
dataset to train a conflict model, which is then merged with the backdoored
model to neutralize malicious behaviors by embedding contradictory information
within the model's parametric memory. Externally, we incorporate convincing
contradictory evidence into the prompt to challenge the model's internal
backdoor knowledge. Experimental results on classification and conversational
tasks across 4 widely used LLMs demonstrate that our method outperforms 8
state-of-the-art backdoor defense baselines. We can reduce the attack success
rate of advanced backdoor attacks by up to 98% while maintaining over 90% clean
data accuracy. Furthermore, our method has proven to be robust against adaptive
backdoor attacks. The code will be open-sourced upon publication.


http://arxiv.org/abs/2411.18269
Hidden Data Privacy Breaches in Federated Learning. (22%)
Xueluan Gong; Yuji Wang; Shuaike Li; Mengyuan Sun; Songze Li; Qian Wang; Kwok-Yan Lam; Chen Chen
  Federated Learning (FL) emerged as a paradigm for conducting machine learning
across broad and decentralized datasets, promising enhanced privacy by
obviating the need for direct data sharing. However, recent studies show that
attackers can steal private data through model manipulation or gradient
analysis. Existing attacks are constrained by low theft quantity or
low-resolution data, and they are often detected through anomaly monitoring in
gradients or weights. In this paper, we propose a novel data-reconstruction
attack leveraging malicious code injection, supported by two key techniques,
i.e., distinctive and sparse encoding design and block partitioning. Unlike
conventional methods that require detectable changes to the model, our method
stealthily embeds a hidden model using parameter sharing to systematically
extract sensitive data. The Fibonacci-based index design ensures efficient,
structured retrieval of memorized data, while the block partitioning method
enhances our method's capability to handle high-resolution images by dividing
them into smaller, manageable units. Extensive experiments on 4 datasets
confirmed that our method is superior to the five state-of-the-art
data-reconstruction attacks under the five respective detection methods. Our
method can handle large-scale and high-resolution data without being detected
or mitigated by state-of-the-art data reconstruction defense methods. In
contrast to baselines, our method can be directly applied to both FedAVG and
FedSGD scenarios, underscoring the need for developers to devise new defenses
against such vulnerabilities. We will open-source our code upon acceptance.


http://arxiv.org/abs/2411.18479
SoK: Watermarking for AI-Generated Content. (3%)
Xuandong Zhao; Sam Gunn; Miranda Christ; Jaiden Fairoze; Andres Fabrega; Nicholas Carlini; Sanjam Garg; Sanghyun Hong; Milad Nasr; Florian Tramer; Somesh Jha; Lei Li; Yu-Xiang Wang; Dawn Song
  As the outputs of generative AI (GenAI) techniques improve in quality, it
becomes increasingly challenging to distinguish them from human-created
content. Watermarking schemes are a promising approach to address the problem
of distinguishing between AI and human-generated content. These schemes embed
hidden signals within AI-generated content to enable reliable detection. While
watermarking is not a silver bullet for addressing all risks associated with
GenAI, it can play a crucial role in enhancing AI safety and trustworthiness by
combating misinformation and deception. This paper presents a comprehensive
overview of watermarking techniques for GenAI, beginning with the need for
watermarking from historical and regulatory perspectives. We formalize the
definitions and desired properties of watermarking schemes and examine the key
objectives and threat models for existing approaches. Practical evaluation
strategies are also explored, providing insights into the development of robust
watermarking techniques capable of resisting various attacks. Additionally, we
review recent representative works, highlight open challenges, and discuss
potential directions for this emerging field. By offering a thorough
understanding of watermarking in GenAI, this work aims to guide researchers in
advancing watermarking methods and applications, and support policymakers in
addressing the broader implications of GenAI.


http://arxiv.org/abs/2411.18207
From Open Vocabulary to Open World: Teaching Vision Language Models to Detect Novel Objects. (1%)
Zizhao Li; Zhengkang Xiang; Joseph West; Kourosh Khoshelham
  Traditional object detection methods operate under the closed-set assumption,
where models can only detect a fixed number of objects predefined in the
training set. Recent works on open vocabulary object detection (OVD) enable the
detection of objects defined by an unbounded vocabulary, which reduces the cost
of training models for specific tasks. However, OVD heavily relies on accurate
prompts provided by an ''oracle'', which limits their use in critical
applications such as driving scene perception. OVD models tend to misclassify
near-out-of-distribution (NOOD) objects that have similar semantics to known
classes, and ignore far-out-of-distribution (FOOD) objects. To address theses
limitations, we propose a framework that enables OVD models to operate in open
world settings, by identifying and incrementally learning novel objects. To
detect FOOD objects, we propose Open World Embedding Learning (OWEL) and
introduce the concept of Pseudo Unknown Embedding which infers the location of
unknown classes in a continuous semantic space based on the information of
known classes. We also propose Multi-Scale Contrastive Anchor Learning (MSCAL),
which enables the identification of misclassified unknown objects by promoting
the intra-class consistency of object embeddings at different scales. The
proposed method achieves state-of-the-art performance in common open world
object detection and autonomous driving benchmarks.


http://arxiv.org/abs/2411.17959
Adversarial Training in Low-Label Regimes with Margin-Based Interpolation. (99%)
Tian Ye; Rajgopal Kannan; Viktor Prasanna
  Adversarial training has emerged as an effective approach to train robust
neural network models that are resistant to adversarial attacks, even in
low-label regimes where labeled data is scarce. In this paper, we introduce a
novel semi-supervised adversarial training approach that enhances both
robustness and natural accuracy by generating effective adversarial examples.
Our method begins by applying linear interpolation between clean and
adversarial examples to create interpolated adversarial examples that cross
decision boundaries by a controlled margin. This sample-aware strategy tailors
adversarial examples to the characteristics of each data point, enabling the
model to learn from the most informative perturbations. Additionally, we
propose a global epsilon scheduling strategy that progressively adjusts the
upper bound of perturbation strengths during training. The combination of these
strategies allows the model to develop increasingly complex decision boundaries
with better robustness and natural accuracy. Empirical evaluations show that
our approach effectively enhances performance against various adversarial
attacks, such as PGD and AutoAttack.


http://arxiv.org/abs/2411.17283
BadScan: An Architectural Backdoor Attack on Visual State Space Models. (98%)
Om Suhas Deshmukh; Sankalp Nagaonkar; Achyut Mani Tripathi; Ashish Mishra
  The newly introduced Visual State Space Model (VMamba), which employs
\textit{State Space Mechanisms} (SSM) to interpret images as sequences of
patches, has shown exceptional performance compared to Vision Transformers
(ViT) across various computer vision tasks. However, recent studies have
highlighted that deep models are susceptible to adversarial attacks. One common
approach is to embed a trigger in the training data to retrain the model,
causing it to misclassify data samples into a target class, a phenomenon known
as a backdoor attack. In this paper, we first evaluate the robustness of the
VMamba model against existing backdoor attacks. Based on this evaluation, we
introduce a novel architectural backdoor attack, termed BadScan, designed to
deceive the VMamba model. This attack utilizes bit plane slicing to create
visually imperceptible backdoored images. During testing, if a trigger is
detected by performing XOR operations between the $k^{th}$ bit planes of the
modified triggered patches, the traditional 2D selective scan (SS2D) mechanism
in the visual state space (VSS) block of VMamba is replaced with our newly
designed BadScan block, which incorporates four newly developed scanning
patterns. We demonstrate that the BadScan backdoor attack represents a
significant threat to visual state space models and remains effective even
after complete retraining from scratch. Experimental results on two widely used
image classification datasets, CIFAR-10, and ImageNet-1K, reveal that while
visual state space models generally exhibit robustness against current backdoor
attacks, the BadScan attack is particularly effective, achieving a higher
Triggered Accuracy Ratio (TAR) in misleading the VMamba model and its variants.


http://arxiv.org/abs/2411.17936
Stealthy Multi-Task Adversarial Attacks. (92%)
Jiacheng Guo; Tianyun Zhang; Lei Li; Haochen Yang; Hongkai Yu; Minghai Qin
  Deep Neural Networks exhibit inherent vulnerabilities to adversarial attacks,
which can significantly compromise their outputs and reliability. While
existing research primarily focuses on attacking single-task scenarios or
indiscriminately targeting all tasks in multi-task environments, we investigate
selectively targeting one task while preserving performance in others within a
multi-task framework. This approach is motivated by varying security priorities
among tasks in real-world applications, such as autonomous driving, where
misinterpreting critical objects (e.g., signs, traffic lights) poses a greater
security risk than minor depth miscalculations. Consequently, attackers may
hope to target security-sensitive tasks while avoiding non-critical tasks from
being compromised, thus evading being detected before compromising crucial
functions. In this paper, we propose a method for the stealthy multi-task
attack framework that utilizes multiple algorithms to inject imperceptible
noise into the input. This novel method demonstrates remarkable efficacy in
compromising the target task while simultaneously maintaining or even enhancing
performance across non-targeted tasks - a criterion hitherto unexplored in the
field. Additionally, we introduce an automated approach for searching the
weighting factors in the loss function, further enhancing attack efficiency.
Experimental results validate our framework's ability to successfully attack
the target task while preserving the performance of non-targeted tasks. The
automated loss function weight searching method demonstrates comparable
efficacy to manual tuning, establishing a state-of-the-art multi-task attack
framework.


http://arxiv.org/abs/2411.17468
Adversarial Bounding Boxes Generation (ABBG) Attack against Visual Object Trackers. (82%)
Fatemeh Nourilenjan Nokabadi; Jean-Francois Lalonde; Christian Gagné
  Adversarial perturbations aim to deceive neural networks into predicting
inaccurate results. For visual object trackers, adversarial attacks have been
developed to generate perturbations by manipulating the outputs. However,
transformer trackers predict a specific bounding box instead of an object
candidate list, which limits the applicability of many existing attack
scenarios. To address this issue, we present a novel white-box approach to
attack visual object trackers with transformer backbones using only one
bounding box. From the tracker predicted bounding box, we generate a list of
adversarial bounding boxes and compute the adversarial loss for those bounding
boxes. Experimental results demonstrate that our simple yet effective attack
outperforms existing attacks against several robust transformer trackers,
including TransT-M, ROMTrack, and MixFormer, on popular benchmark tracking
datasets such as GOT-10k, UAV123, and VOT2022STS.


http://arxiv.org/abs/2411.18648
MADE: Graph Backdoor Defense with Masked Unlearning. (82%)
Xiao Lin; Mingjie Li; Yisen Wang
  Graph Neural Networks (GNNs) have garnered significant attention from
researchers due to their outstanding performance in handling graph-related
tasks, such as social network analysis, protein design, and so on. Despite
their widespread application, recent research has demonstrated that GNNs are
vulnerable to backdoor attacks, implemented by injecting triggers into the
training datasets. Trained on the poisoned data, GNNs will predict target
labels when attaching trigger patterns to inputs. This vulnerability poses
significant security risks for applications of GNNs in sensitive domains, such
as drug discovery. While there has been extensive research into backdoor
defenses for images, strategies to safeguard GNNs against such attacks remain
underdeveloped. Furthermore, we point out that conventional backdoor defense
methods designed for images cannot work well when directly implemented on graph
data. In this paper, we first analyze the key difference between image backdoor
and graph backdoor attacks. Then we tackle the graph defense problem by
presenting a novel approach called MADE, which devises an adversarial mask
generation mechanism that selectively preserves clean sub-graphs and further
leverages masks on edge weights to eliminate the influence of triggers
effectively. Extensive experiments across various graph classification tasks
demonstrate the effectiveness of MADE in significantly reducing the attack
success rate (ASR) while maintaining a high classification accuracy.


http://arxiv.org/abs/2411.18000
Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models. (75%)
Shuyang Hao; Bryan Hooi; Jun Liu; Kai-Wei Chang; Zi Huang; Yujun Cai
  Despite inheriting security measures from underlying language models,
Vision-Language Models (VLMs) may still be vulnerable to safety alignment
issues. Through empirical analysis, we uncover two critical findings:
scenario-matched images can significantly amplify harmful outputs, and contrary
to common assumptions in gradient-based attacks, minimal loss values do not
guarantee optimal attack effectiveness. Building on these insights, we
introduce MLAI (Multi-Loss Adversarial Images), a novel jailbreak framework
that leverages scenario-aware image generation for semantic alignment, exploits
flat minima theory for robust adversarial image selection, and employs
multi-image collaborative attacks for enhanced effectiveness. Extensive
experiments demonstrate MLAI's significant impact, achieving attack success
rates of 77.75% on MiniGPT-4 and 82.80% on LLaVA-2, substantially outperforming
existing methods by margins of 34.37% and 12.77% respectively. Furthermore,
MLAI shows considerable transferability to commercial black-box VLMs, achieving
up to 60.11% success rate. Our work reveals fundamental visual vulnerabilities
in current VLMs safety mechanisms and underscores the need for stronger
defenses. Warning: This paper contains potentially harmful example text.


http://arxiv.org/abs/2411.18027
Privacy-preserving Robotic-based Multi-factor Authentication Scheme for Secure Automated Delivery System. (9%)
Yang Yang; Aryan Mohammadi Pasikhani; Prosanta Gope; Biplab Sikdar
  Package delivery is a critical aspect of various industries, but it often
incurs high financial costs and inefficiencies when relying solely on human
resources. The last-mile transport problem, in particular, contributes
significantly to the expenditure of human resources in major companies.
Robot-based delivery systems have emerged as a potential solution for last-mile
delivery to address this challenge. However, robotic delivery systems still
face security and privacy issues, like impersonation, replay, man-in-the-middle
attacks (MITM), unlinkability, and identity theft. In this context, we propose
a privacy-preserving multi-factor authentication scheme specifically designed
for robot delivery systems. Additionally, AI-assisted robotic delivery systems
are susceptible to machine learning-based attacks (e.g. FGSM, PGD, etc.). We
introduce the \emph{first} transformer-based audio-visual fusion defender to
tackle this issue, which effectively provides resilience against adversarial
samples. Furthermore, we provide a rigorous formal analysis of the proposed
protocol and also analyse the protocol security using a popular symbolic proof
tool called ProVerif and Scyther. Finally, we present a real-world
implementation of the proposed robotic system with the computation cost and
energy consumption analysis. Code and pre-trained models are available at:
https://drive.google.com/drive/folders/18B2YbxtV0Pyj5RSFX-ZzCGtFOyorBHil


http://arxiv.org/abs/2411.17453
PEFTGuard: Detecting Backdoor Attacks Against Parameter-Efficient Fine-Tuning. (2%)
Zhen Sun; Tianshuo Cong; Yule Liu; Chenhao Lin; Xinlei He; Rongmao Chen; Xingshuo Han; Xinyi Huang
  Fine-tuning is an essential process to improve the performance of Large
Language Models (LLMs) in specific domains, with Parameter-Efficient
Fine-Tuning (PEFT) gaining popularity due to its capacity to reduce
computational demands through the integration of low-rank adapters. These
lightweight adapters, such as LoRA, can be shared and utilized on open-source
platforms. However, adversaries could exploit this mechanism to inject
backdoors into these adapters, resulting in malicious behaviors like incorrect
or harmful outputs, which pose serious security risks to the community.
Unfortunately, few of the current efforts concentrate on analyzing the backdoor
patterns or detecting the backdoors in the adapters.
  To fill this gap, we first construct (and will release) PADBench, a
comprehensive benchmark that contains 13,300 benign and backdoored adapters
fine-tuned with various datasets, attack strategies, PEFT methods, and LLMs.
Moreover, we propose PEFTGuard, the first backdoor detection framework against
PEFT-based adapters. Extensive evaluation upon PADBench shows that PEFTGuard
outperforms existing detection methods, achieving nearly perfect detection
accuracy (100%) in most cases. Notably, PEFTGuard exhibits zero-shot
transferability on three aspects, including different attacks, PEFT methods,
and adapter ranks. In addition, we consider various adaptive attacks to
demonstrate the high robustness of PEFTGuard. We further explore several
possible backdoor mitigation defenses, finding fine-mixing to be the most
effective method. We envision our benchmark and method can shed light on future
LLM backdoor detection research.


http://arxiv.org/abs/2411.17585
Multi-Objective Reinforcement Learning for Automated Resilient Cyber Defence. (1%)
Ross O'Driscoll; Claudia Hagen; Joe Bater; James M. Adams
  Cyber-attacks pose a security threat to military command and control
networks, Intelligence, Surveillance, and Reconnaissance (ISR) systems, and
civilian critical national infrastructure. The use of artificial intelligence
and autonomous agents in these attacks increases the scale, range, and
complexity of this threat and the subsequent disruption they cause. Autonomous
Cyber Defence (ACD) agents aim to mitigate this threat by responding at machine
speed and at the scale required to address the problem. Sequential
decision-making algorithms such as Deep Reinforcement Learning (RL) provide a
promising route to create ACD agents. These algorithms focus on a single
objective such as minimizing the intrusion of red agents on the network, by
using a handcrafted weighted sum of rewards. This approach removes the ability
to adapt the model during inference, and fails to address the many competing
objectives present when operating and protecting these networks. Conflicting
objectives, such as restoring a machine from a back-up image, must be carefully
balanced with the cost of associated down-time, or the disruption to network
traffic or services that might result. Instead of pursing a Single-Objective RL
(SORL) approach, here we present a simple example of a multi-objective network
defence game that requires consideration of both defending the network against
red-agents and maintaining critical functionality of green-agents. Two
Multi-Objective Reinforcement Learning (MORL) algorithms, namely
Multi-Objective Proximal Policy Optimization (MOPPO), and Pareto-Conditioned
Networks (PCN), are used to create two trained ACD agents whose performance is
compared on our Multi-Objective Cyber Defence game. The benefits and
limitations of MORL ACD agents in comparison to SORL ACD agents are discussed
based on the investigations of this game.


http://arxiv.org/abs/2411.18028
Improved Parallel Derandomization via Finite Automata with Applications. (1%)
Jeff Giliberti; David G. Harris
  One main genre of algorithmic derandomization comes from the construction of
probability distributions with small support that fool a randomized algorithm.
This is especially well-suited to parallelization, i.e. NC algorithms. A
significant abstraction of these methods can be formulated in terms of fooling
polynomial-space statistical tests computed via finite automata (Sivakumar
2002); this encompasses $k$-wise independence, sums of random variables, and
many other properties.
  We describe new parallel algorithms to fool general finite-state automata
with significantly reduced processor complexity. The analysis is also
simplified because we can cleanly separate the problem-specific optimizations
from the general lattice discrepancy problems at the core of the
automaton-fooling construction. We illustrate with improved applications to the
Gale-Berlekamp Switching Game and to approximate MAX-CUT via SDP rounding.


http://arxiv.org/abs/2411.16598
Unlocking The Potential of Adaptive Attacks on Diffusion-Based Purification. (99%)
Andre Kassis; Urs Hengartner; Yaoliang Yu
  Diffusion-based purification (DBP) is a defense against adversarial examples
(AEs), amassing popularity for its ability to protect classifiers in an
attack-oblivious manner and resistance to strong adversaries with access to the
defense. Its robustness has been claimed to ensue from the reliance on
diffusion models (DMs) that project the AEs onto the natural distribution. We
revisit this claim, focusing on gradient-based strategies that back-propagate
the loss gradients through the defense, commonly referred to as ``adaptive
attacks". Analytically, we show that such an optimization method invalidates
DBP's core foundations, effectively targeting the DM rather than the classifier
and restricting the purified outputs to a distribution over malicious samples
instead. Thus, we reassess the reported empirical robustness, uncovering
implementation flaws in the gradient back-propagation techniques used thus far
for DBP. We fix these issues, providing the first reliable gradient library for
DBP and demonstrating how adaptive attacks drastically degrade its robustness.
We then study a less efficient yet stricter majority-vote setting where the
classifier evaluates multiple purified copies of the input to make its
decision. Here, DBP's stochasticity enables it to remain partially robust
against traditional norm-bounded AEs. We propose a novel adaptation of a recent
optimization method against deepfake watermarking that crafts systemic
malicious perturbations while ensuring imperceptibility. When integrated with
the adaptive attack, it completely defeats DBP, even in the majority-vote
setup. Our findings prove that DBP, in its current state, is not a viable
defense against AEs.


http://arxiv.org/abs/2411.16622
Imperceptible Adversarial Examples in the Physical World. (99%)
Weilin Xu; Sebastian Szyller; Cory Cornelius; Luis Murillo Rojas; Marius Arvinte; Alvaro Velasquez; Jason Martin; Nageen Himayat
  Adversarial examples in the digital domain against deep learning-based
computer vision models allow for perturbations that are imperceptible to human
eyes. However, producing similar adversarial examples in the physical world has
been difficult due to the non-differentiable image distortion functions in
visual sensing systems. The existing algorithms for generating physically
realizable adversarial examples often loosen their definition of adversarial
examples by allowing unbounded perturbations, resulting in obvious or even
strange visual patterns. In this work, we make adversarial examples
imperceptible in the physical world using a straight-through estimator (STE,
a.k.a. BPDA). We employ STE to overcome the non-differentiability -- applying
exact, non-differentiable distortions in the forward pass of the
backpropagation step, and using the identity function in the backward pass. Our
differentiable rendering extension to STE also enables imperceptible
adversarial patches in the physical world. Using printout photos, and
experiments in the CARLA simulator, we show that STE enables fast generation of
$\ell_\infty$ bounded adversarial examples despite the non-differentiable
distortions. To the best of our knowledge, this is the first work demonstrating
imperceptible adversarial examples bounded by small $\ell_\infty$ norms in the
physical world that force zero classification accuracy in the global
perturbation threat model and cause near-zero ($4.22\%$) AP50 in object
detection in the patch perturbation threat model. We urge the community to
re-evaluate the threat of adversarial examples in the physical world.


http://arxiv.org/abs/2411.16782
Scaling Laws for Black box Adversarial Attacks. (99%)
Chuan Liu; Huanran Chen; Yichi Zhang; Yinpeng Dong; Jun Zhu
  A longstanding problem of deep learning models is their vulnerability to
adversarial examples, which are often generated by applying imperceptible
perturbations to natural examples. Adversarial examples exhibit cross-model
transferability, enabling to attack black-box models with limited information
about their architectures and parameters. Model ensembling is an effective
strategy to improve the transferability by attacking multiple surrogate models
simultaneously. However, as prior studies usually adopt few models in the
ensemble, there remains an open question of whether scaling the number of
models can further improve black-box attacks. Inspired by the findings in large
foundation models, we investigate the scaling laws of black-box adversarial
attacks in this work. By analyzing the relationship between the number of
surrogate models and transferability of adversarial examples, we conclude with
clear scaling laws, emphasizing the potential of using more surrogate models to
enhance adversarial transferability. Extensive experiments verify the claims on
standard image classifiers, multimodal large language models, and even
proprietary models like GPT-4o, demonstrating consistent scaling effects and
impressive attack success rates with more surrogate models. Further studies by
visualization indicate that scaled attacks bring better interpretability in
semantics, indicating that the common features of models are captured.


http://arxiv.org/abs/2411.16437
Privacy Protection in Personalized Diffusion Models via Targeted Cross-Attention Adversarial Attack. (81%)
Xide Xu; Muhammad Atif Butt; Sandesh Kamath; Bogdan Raducanu
  The growing demand for customized visual content has led to the rise of
personalized text-to-image (T2I) diffusion models. Despite their remarkable
potential, they pose significant privacy risk when misused for malicious
purposes. In this paper, we propose a novel and efficient adversarial attack
method, Concept Protection by Selective Attention Manipulation (CoPSAM) which
targets only the cross-attention layers of a T2I diffusion model. For this
purpose, we carefully construct an imperceptible noise to be added to clean
samples to get their adversarial counterparts. This is obtained during the
fine-tuning process by maximizing the discrepancy between the corresponding
cross-attention maps of the user-specific token and the class-specific token,
respectively. Experimental validation on a subset of CelebA-HQ face images
dataset demonstrates that our approach outperforms existing methods. Besides
this, our method presents two important advantages derived from the qualitative
evaluation: (i) we obtain better protection results for lower noise levels than
our competitors; and (ii) we protect the content from unauthorized use thereby
protecting the individual's identity from potential misuse.


http://arxiv.org/abs/2411.17746
UVCG: Leveraging Temporal Consistency for Universal Video Protection. (54%)
KaiZhou Li; Jindong Gu; Xinchun Yu; Junjie Cao; Yansong Tang; Xiao-Ping Zhang
  The security risks of AI-driven video editing have garnered significant
attention. Although recent studies indicate that adding perturbations to images
can protect them from malicious edits, directly applying image-based methods to
perturb each frame in a video becomes ineffective, as video editing techniques
leverage the consistency of inter-frame information to restore individually
perturbed content. To address this challenge, we leverage the temporal
consistency of video content to propose a straightforward and efficient, yet
highly effective and broadly applicable approach, Universal Video Consistency
Guard (UVCG). UVCG embeds the content of another video(target video) within a
protected video by introducing continuous, imperceptible perturbations which
has the ability to force the encoder of editing models to map continuous inputs
to misaligned continuous outputs, thereby inhibiting the generation of videos
consistent with the intended textual prompts. Additionally leveraging
similarity in perturbations between adjacent frames, we improve the
computational efficiency of perturbation generation by employing a
perturbation-reuse strategy. We applied UVCG across various versions of Latent
Diffusion Models (LDM) and assessed its effectiveness and generalizability
across multiple LDM-based editing pipelines. The results confirm the
effectiveness, transferability, and efficiency of our approach in safeguarding
video content from unauthorized modifications.


http://arxiv.org/abs/2411.16512
Guarding the Gate: ConceptGuard Battles Concept-Level Backdoors in Concept Bottleneck Models. (50%)
Songning Lai; Yu Huang; Jiayu Yang; Gaoxiang Huang; Wenshuo Chen; Yutao Yue
  The increasing complexity of AI models, especially in deep learning, has
raised concerns about transparency and accountability, particularly in
high-stakes applications like medical diagnostics, where opaque models can
undermine trust. Explainable Artificial Intelligence (XAI) aims to address
these issues by providing clear, interpretable models. Among XAI techniques,
Concept Bottleneck Models (CBMs) enhance transparency by using high-level
semantic concepts. However, CBMs are vulnerable to concept-level backdoor
attacks, which inject hidden triggers into these concepts, leading to
undetectable anomalous behavior. To address this critical security gap, we
introduce ConceptGuard, a novel defense framework specifically designed to
protect CBMs from concept-level backdoor attacks. ConceptGuard employs a
multi-stage approach, including concept clustering based on text distance
measurements and a voting mechanism among classifiers trained on different
concept subgroups, to isolate and mitigate potential triggers. Our
contributions are threefold: (i) we present ConceptGuard as the first defense
mechanism tailored for concept-level backdoor attacks in CBMs; (ii) we provide
theoretical guarantees that ConceptGuard can effectively defend against such
attacks within a certain trigger size threshold, ensuring robustness; and (iii)
we demonstrate that ConceptGuard maintains the high performance and
interpretability of CBMs, crucial for trustworthiness. Through comprehensive
experiments and theoretical proofs, we show that ConceptGuard significantly
enhances the security and trustworthiness of CBMs, paving the way for their
secure deployment in critical applications.


http://arxiv.org/abs/2411.16832
Edit Away and My Face Will not Stay: Personal Biometric Defense against Malicious Generative Editing. (50%)
Hanhui Wang; Yihua Zhang; Ruizheng Bai; Yue Zhao; Sijia Liu; Zhengzhong Tu
  Recent advancements in diffusion models have made generative image editing
more accessible, enabling creative edits but raising ethical concerns,
particularly regarding malicious edits to human portraits that threaten privacy
and identity security. Existing protection methods primarily rely on
adversarial perturbations to nullify edits but often fail against diverse
editing requests. We propose FaceLock, a novel approach to portrait protection
that optimizes adversarial perturbations to destroy or significantly alter
biometric information, rendering edited outputs biometrically unrecognizable.
FaceLock integrates facial recognition and visual perception into perturbation
optimization to provide robust protection against various editing attempts. We
also highlight flaws in commonly used evaluation metrics and reveal how they
can be manipulated, emphasizing the need for reliable assessments of
protection. Experiments show FaceLock outperforms baselines in defending
against malicious edits and is robust against purification techniques. Ablation
studies confirm its stability and broad applicability across diffusion-based
editing algorithms. Our work advances biometric defense and sets the foundation
for privacy-preserving practices in image editing. The code is available at:
https://github.com/taco-group/FaceLock.


http://arxiv.org/abs/2411.16162
Sparse patches adversarial attacks via extrapolating point-wise information. (47%)
Yaniv Nemcovsky; Avi Mendelson; Chaim Baskin
  Sparse and patch adversarial attacks were previously shown to be applicable
in realistic settings and are considered a security risk to autonomous systems.
Sparse adversarial perturbations constitute a setting in which the adversarial
perturbations are limited to affecting a relatively small number of points in
the input. Patch adversarial attacks denote the setting where the sparse
attacks are limited to a given structure, i.e., sparse patches with a given
shape and number. However, previous patch adversarial attacks do not
simultaneously optimize multiple patches' locations and perturbations. This
work suggests a novel approach for sparse patches adversarial attacks via
point-wise trimming dense adversarial perturbations. Our approach enables
simultaneous optimization of multiple sparse patches' locations and
perturbations for any given number and shape. Moreover, our approach is also
applicable for standard sparse adversarial attacks, where we show that it
significantly improves the state-of-the-art over multiple extensive settings. A
reference implementation of the proposed method and the reported experiments is
provided at \url{https://github.com/yanemcovsky/SparsePatches.git}


http://arxiv.org/abs/2411.16154
DeDe: Detecting Backdoor Samples for SSL Encoders via Decoders. (10%)
Sizai Hou; Songze Li; Duanyi Yao
  Self-supervised learning (SSL) is pervasively exploited in training
high-quality upstream encoders with a large amount of unlabeled data. However,
it is found to be susceptible to backdoor attacks merely via polluting a small
portion of training data. The victim encoders mismatch triggered inputs with
target embeddings, e.g., match the triggered cat input to an airplane
embedding, such that the downstream tasks are affected to misbehave when the
trigger is activated. Emerging backdoor attacks have shown great threats in
different SSL paradigms such as contrastive learning and CLIP, while few
research is devoted to defending against such attacks. Besides, the existing
ones fall short in detecting advanced stealthy backdoors. To address the
limitations, we propose a novel detection mechanism, DeDe, which detects the
activation of the backdoor mapping with the cooccurrence of victim encoder and
trigger inputs. Specifically, DeDe trains a decoder for the SSL encoder on an
auxiliary dataset (can be out-of-distribution or even slightly poisoned), such
that for any triggered input that misleads to the target embedding, the decoder
outputs an image significantly different from the input. We empirically
evaluate DeDe on both contrastive learning and CLIP models against various
types of backdoor attacks, and demonstrate its superior performance over SOTA
detection methods in both upstream detection performance and ability of
preventing backdoors in downstream tasks.


http://arxiv.org/abs/2411.17026
RED: Robust Environmental Design. (10%)
Jinghan Yang
  The classification of road signs by autonomous systems, especially those
reliant on visual inputs, is highly susceptible to adversarial attacks.
Traditional approaches to mitigating such vulnerabilities have focused on
enhancing the robustness of classification models. In contrast, this paper
adopts a fundamentally different strategy aimed at increasing robustness
through the redesign of road signs themselves. We propose an attacker-agnostic
learning scheme to automatically design road signs that are robust to a wide
array of patch-based attacks. Empirical tests conducted in both digital and
physical environments demonstrate that our approach significantly reduces
vulnerability to patch attacks, outperforming existing techniques.


http://arxiv.org/abs/2411.16167
BadSFL: Backdoor Attack against Scaffold Federated Learning. (3%)
Xingshuo Han; Xuanye Zhang; Xiang Lan; Haozhao Wang; Shengmin Xu; Shen Ren; Jason Zeng; Ming Wu; Michael Heinrich; Tianwei Zhang
  Federated learning (FL) enables the training of deep learning models on
distributed clients to preserve data privacy. However, this learning paradigm
is vulnerable to backdoor attacks, where malicious clients can upload poisoned
local models to embed backdoors into the global model, leading to
attacker-desired predictions. Existing backdoor attacks mainly focus on FL with
independently and identically distributed (IID) scenarios, while real-world FL
training data are typically non-IID. Current strategies for non-IID backdoor
attacks suffer from limitations in maintaining effectiveness and durability. To
address these challenges, we propose a novel backdoor attack method, BadSFL,
specifically designed for the FL framework using the scaffold aggregation
algorithm in non-IID settings. BadSFL leverages a Generative Adversarial
Network (GAN) based on the global model to complement the training set,
achieving high accuracy on both backdoor and benign samples. It utilizes a
specific feature as the backdoor trigger to ensure stealthiness, and exploits
the Scaffold's control variate to predict the global model's convergence
direction, ensuring the backdoor's persistence. Extensive experiments on three
benchmark datasets demonstrate the high effectiveness, stealthiness, and
durability of BadSFL. Notably, our attack remains effective over 60 rounds in
the global model and up to 3 times longer than existing baseline attacks after
stopping the injection of malicious updates.


http://arxiv.org/abs/2411.16120
Why the Agent Made that Decision: Explaining Deep Reinforcement Learning with Vision Masks. (2%)
Rui Zuo; Zifan Wang; Simon Khan; Garrett Ethan Katz; Qinru Qiu
  Due to the inherent lack of transparency in deep neural networks, it is
challenging for deep reinforcement learning (DRL) agents to gain trust and
acceptance from users, especially in safety-critical applications such as
medical diagnosis and military operations. Existing methods for explaining an
agent's decision either require to retrain the agent using models that support
explanation generation or rely on perturbation-based techniques to reveal the
significance of different input features in the decision making process.
However, retraining the agent may compromise its integrity and performance,
while perturbation-based methods have limited performance and lack knowledge
accumulation or learning capabilities. Moreover, since each perturbation is
performed independently, the joint state of the perturbed inputs may not be
physically meaningful. To address these challenges, we introduce
$\textbf{VisionMask}$, a standalone explanation model trained end-to-end to
identify the most critical regions in the agent's visual input that can explain
its actions. VisionMask is trained in a self-supervised manner without relying
on human-generated labels. Importantly, its training does not alter the agent
model, hence preserving the agent's performance and integrity. We evaluate
VisionMask on Super Mario Bros (SMB) and three Atari games. Compared to
existing methods, VisionMask achieves a 14.9% higher insertion accuracy and a
30.08% higher F1-Score in reproducing original actions from the selected visual
explanations. We also present examples illustrating how VisionMask can be used
for counterfactual analysis.


http://arxiv.org/abs/2411.16817
XAI and Android Malware Models. (2%)
Maithili Kulkarni; Mark Stamp
  Android malware detection based on machine learning (ML) and deep learning
(DL) models is widely used for mobile device security. Such models offer
benefits in terms of detection accuracy and efficiency, but it is often
difficult to understand how such learning models make decisions. As a result,
these popular malware detection strategies are generally treated as black
boxes, which can result in a lack of trust in the decisions made, as well as
making adversarial attacks more difficult to detect. The field of eXplainable
Artificial Intelligence (XAI) attempts to shed light on such black box models.
In this paper, we apply XAI techniques to ML and DL models that have been
trained on a challenging Android malware classification problem. Specifically,
the classic ML models considered are Support Vector Machines (SVM), Random
Forest, and $k$-Nearest Neighbors ($k$-NN), while the DL models we consider are
Multi-Layer Perceptrons (MLP) and Convolutional Neural Networks (CNN). The
state-of-the-art XAI techniques that we apply to these trained models are Local
Interpretable Model-agnostic Explanations (LIME), Shapley Additive exPlanations
(SHAP), PDP plots, ELI5, and Class Activation Mapping (CAM). We obtain global
and local explanation results, and we discuss the utility of XAI techniques in
this problem domain. We also provide a literature review of XAI work related to
Android malware.


http://arxiv.org/abs/2411.16148
Revisiting Marr in Face: The Building of 2D--2.5D--3D Representations in Deep Neural Networks. (1%)
Xiangyu Zhu; Chang Yu; Jiankuo Zhao; Zhaoxiang Zhang; Stan Z. Li; Zhen Lei
  David Marr's seminal theory of vision proposes that the human visual system
operates through a sequence of three stages, known as the 2D sketch, the 2.5D
sketch, and the 3D model. In recent years, Deep Neural Networks (DNN) have been
widely thought to have reached a level comparable to human vision. However, the
mechanisms by which DNNs accomplish this and whether they adhere to Marr's
2D--2.5D--3D construction theory remain unexplored. In this paper, we delve
into the perception task to explore these questions and find evidence
supporting Marr's theory. We introduce a graphics probe, a sub-network crafted
to reconstruct the original image from the network's intermediate layers. The
key to the graphics probe is its flexible architecture that supports image in
both 2D and 3D formats, as well as in a transitional state between them. By
injecting graphics probes into neural networks, and analyzing their behavior in
reconstructing images, we find that DNNs initially encode images as 2D
representations in low-level layers, and finally construct 3D representations
in high-level layers. Intriguingly, in mid-level layers, DNNs exhibit a hybrid
state, building a geometric representation that s sur normals within a narrow
depth range, akin to the appearance of a low-relief sculpture. This stage
resembles the 2.5D representations, providing a view of how DNNs evolve from 2D
to 3D in the perception process. The graphics probe therefore serves as a tool
for peering into the mechanisms of DNN, providing empirical support for Marr's
theory.


http://arxiv.org/abs/2411.15720
Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks. (99%)
Peng Xie; Yequan Bie; Jianda Mao; Yangqiu Song; Yang Wang; Hao Chen; Kani Chen
  Pre-trained vision-language models (VLMs) have showcased remarkable
performance in image and natural language understanding, such as image
captioning and response generation. As the practical applications of
vision-language models become increasingly widespread, their potential safety
and robustness issues raise concerns that adversaries may evade the system and
cause these models to generate toxic content through malicious attacks.
Therefore, evaluating the robustness of open-source VLMs against adversarial
attacks has garnered growing attention, with transfer-based attacks as a
representative black-box attacking strategy. However, most existing
transfer-based attacks neglect the importance of the semantic correlations
between vision and text modalities, leading to sub-optimal adversarial example
generation and attack performance. To address this issue, we present Chain of
Attack (CoA), which iteratively enhances the generation of adversarial examples
based on the multi-modal semantic update using a series of intermediate
attacking steps, achieving superior adversarial transferability and efficiency.
A unified attack success rate computing method is further proposed for
automatic evasion evaluation. Extensive experiments conducted under the most
realistic and high-stakes scenario, demonstrate that our attacking strategy can
effectively mislead models to generate targeted responses using only black-box
attacks without any knowledge of the victim models. The comprehensive
robustness evaluation in our paper provides insight into the vulnerabilities of
VLMs and offers a reference for the safety considerations of future model
developments.


http://arxiv.org/abs/2411.15878
ExAL: An Exploration Enhanced Adversarial Learning Algorithm. (92%)
A Vinil; Aneesh Sreevallabh Chivukula; Pranav Chintareddy
  Adversarial learning is critical for enhancing model robustness, aiming to
defend against adversarial attacks that jeopardize machine learning systems.
Traditional methods often lack efficient mechanisms to explore diverse
adversarial perturbations, leading to limited model resilience. Inspired by
game-theoretic principles, where adversarial dynamics are analyzed through
frameworks like Nash equilibrium, exploration mechanisms in such setups allow
for the discovery of diverse strategies, enhancing system robustness. However,
existing adversarial learning methods often fail to incorporate structured
exploration effectively, reducing their ability to improve model defense
comprehensively. To address these challenges, we propose a novel
Exploration-enhanced Adversarial Learning Algorithm (ExAL), leveraging the
Exponentially Weighted Momentum Particle Swarm Optimizer (EMPSO) to generate
optimized adversarial perturbations. ExAL integrates exploration-driven
mechanisms to discover perturbations that maximize impact on the model's
decision boundary while preserving structural coherence in the data. We
evaluate the performance of ExAL on the MNIST Handwritten Digits and Blended
Malware datasets. Experimental results demonstrate that ExAL significantly
enhances model resilience to adversarial attacks by improving robustness
through adversarial learning.


http://arxiv.org/abs/2411.15921
A Tunable Despeckling Neural Network Stabilized via Diffusion Equation. (64%)
Yi Ran; Zhichang Guo; Jia Li; Yao Li; Martin Burger; Boying Wu
  The removal of multiplicative Gamma noise is a critical research area in the
application of synthetic aperture radar (SAR) imaging, where neural networks
serve as a potent tool. However, real-world data often diverges from
theoretical models, exhibiting various disturbances, which makes the neural
network less effective. Adversarial attacks can be used as a criterion for
judging the adaptability of neural networks to real data, since adversarial
attacks can find the most extreme perturbations that make neural networks
ineffective. In this work, the diffusion equation is designed as a
regularization block to provide sufficient regularity to the whole neural
network, due to its spontaneous dissipative nature. We propose a tunable,
regularized neural network framework that unrolls a shallow denoising neural
network block and a diffusion regularity block into a single network for
end-to-end training. The linear heat equation, known for its inherent
smoothness and low-pass filtering properties, is adopted as the diffusion
regularization block. In our model, a single time step hyperparameter governs
the smoothness of the outputs and can be adjusted dynamically, significantly
enhancing flexibility. The stability and convergence of our model are
theoretically proven. Experimental results demonstrate that the proposed model
effectively eliminates high-frequency oscillations induced by adversarial
attacks. Finally, the proposed model is benchmarked against several
state-of-the-art denoising methods on simulated images, adversarial samples,
and real SAR images, achieving superior performance in both quantitative and
visual evaluations.


http://arxiv.org/abs/2411.16763
Hide in Plain Sight: Clean-Label Backdoor for Auditing Membership Inference. (10%)
Depeng Chen; Hao Chen; Hulin Jin; Jie Cui; Hong Zhong
  Membership inference attacks (MIAs) are critical tools for assessing privacy
risks and ensuring compliance with regulations like the General Data Protection
Regulation (GDPR). However, their potential for auditing unauthorized use of
data remains under explored. To bridge this gap, we propose a novel clean-label
backdoor-based approach for MIAs, designed specifically for robust and stealthy
data auditing. Unlike conventional methods that rely on detectable poisoned
samples with altered labels, our approach retains natural labels, enhancing
stealthiness even at low poisoning rates. Our approach employs an optimal
trigger generated by a shadow model that mimics the target model's behavior.
This design minimizes the feature-space distance between triggered samples and
the source class while preserving the original data labels. The result is a
powerful and undetectable auditing mechanism that overcomes limitations of
existing approaches, such as label inconsistencies and visual artifacts in
poisoned samples. The proposed method enables robust data auditing through
black-box access, achieving high attack success rates across diverse datasets
and model architectures. Additionally, it addresses challenges related to
trigger stealthiness and poisoning durability, establishing itself as a
practical and effective solution for data auditing. Comprehensive experiments
validate the efficacy and generalizability of our approach, outperforming
several baseline methods in both stealth and attack success metrics.


http://arxiv.org/abs/2411.16024
Stealth Attacks Against Moving Target Defense for Smart Grid. (2%)
Ke Sun; Iñaki Esnaola; H. Vincent Poor
  Data injection attacks (DIAs) pose a significant cybersecurity threat to the
Smart Grid by enabling an attacker to compromise the integrity of data
acquisition and manipulate estimated states without triggering bad data
detection procedures. To mitigate this vulnerability, the moving target defense
(MTD) alters branch admittances to mismatch the system information that is
available to an attacker, thereby inducing an imperfect DIA construction that
results in degradation of attack performance. In this paper, we first analyze
the existence of stealth attacks for the case in which the MTD strategy only
changes the admittance of a single branch. Equipped with this initial insight,
we then extend the results to the case in which multiple branches are protected
by the MTD strategy. Remarkably, we show that stealth attacks can be
constructed with information only about which branches are protected, without
knowledge about the particular admittance value changes. Furthermore, we
provide a sufficient protection condition for the MTD strategy via
graph-theoretic tools that guarantee that the system is not vulnerable to DIAs.
Numerical simulations are implemented on IEEE test systems to validate the
obtained results.


http://arxiv.org/abs/2411.15976
DRIVE: Dual-Robustness via Information Variability and Entropic Consistency in Source-Free Unsupervised Domain Adaptation. (2%)
Ruiqiang Xiao; Songning Lai; Yijun Yang; Jiemin Wu; Yutao Yue; Lei Zhu
  Adapting machine learning models to new domains without labeled data,
especially when source data is inaccessible, is a critical challenge in
applications like medical imaging, autonomous driving, and remote sensing. This
task, known as Source-Free Unsupervised Domain Adaptation (SFUDA), involves
adapting a pre-trained model to a target domain using only unlabeled target
data, which can lead to issues such as overfitting, underfitting, and poor
generalization due to domain discrepancies and noise. Existing SFUDA methods
often rely on single-model architectures, struggling with uncertainty and
variability in the target domain. To address these challenges, we propose DRIVE
(Dual-Robustness through Information Variability and Entropy), a novel SFUDA
framework leveraging a dual-model architecture. The two models, initialized
with identical weights, work in parallel to capture diverse target domain
characteristics. One model is exposed to perturbations via projection gradient
descent (PGD) guided by mutual information, focusing on high-uncertainty
regions. We also introduce an entropy-aware pseudo-labeling strategy that
adjusts label weights based on prediction uncertainty, ensuring the model
focuses on reliable data while avoiding noisy regions. The adaptation process
has two stages: the first aligns the models on stable features using a mutual
information consistency loss, and the second dynamically adjusts the
perturbation level based on the loss from the first stage, encouraging the
model to explore a broader range of the target domain while preserving existing
performance. This enhances generalization capabilities and robustness against
interference. Evaluations on standard SFUDA benchmarks show that DRIVE
consistently outperforms previous methods, delivering improved adaptation
accuracy and stability across complex target domains.


http://arxiv.org/abs/2411.15553
Improving Transferable Targeted Attacks with Feature Tuning Mixup. (99%)
Kaisheng Liang; Xuelong Dai; Yanjie Li; Dong Wang; Bin Xiao
  Deep neural networks exhibit vulnerability to adversarial examples that can
transfer across different models. A particularly challenging problem is
developing transferable targeted attacks that can mislead models into
predicting specific target classes. While various methods have been proposed to
enhance attack transferability, they often incur substantial computational
costs while yielding limited improvements. Recent clean feature mixup methods
use random clean features to perturb the feature space but lack optimization
for disrupting adversarial examples, overlooking the advantages of
attack-specific perturbations. In this paper, we propose Feature Tuning Mixup
(FTM), a novel method that enhances targeted attack transferability by
combining both random and optimized noises in the feature space. FTM introduces
learnable feature perturbations and employs an efficient stochastic update
strategy for optimization. These learnable perturbations facilitate the
generation of more robust adversarial examples with improved transferability.
We further demonstrate that attack performance can be enhanced through an
ensemble of multiple FTM-perturbed surrogate models. Extensive experiments on
the ImageNet-compatible dataset across various models demonstrate that our
method achieves significant improvements over state-of-the-art methods while
maintaining low computational cost.


http://arxiv.org/abs/2411.15555
Enhancing the Transferability of Adversarial Attacks on Face Recognition with Diverse Parameters Augmentation. (99%)
Fengfan Zhou; Bangjie Yin; Hefei Ling; Qianyu Zhou; Wenxuan Wang
  Face Recognition (FR) models are vulnerable to adversarial examples that
subtly manipulate benign face images, underscoring the urgent need to improve
the transferability of adversarial attacks in order to expose the blind spots
of these systems. Existing adversarial attack methods often overlook the
potential benefits of augmenting the surrogate model with diverse
initializations, which limits the transferability of the generated adversarial
examples. To address this gap, we propose a novel method called Diverse
Parameters Augmentation (DPA) attack method, which enhances surrogate models by
incorporating diverse parameter initializations, resulting in a broader and
more diverse set of surrogate models. Specifically, DPA consists of two key
stages: Diverse Parameters Optimization (DPO) and Hard Model Aggregation (HMA).
In the DPO stage, we initialize the parameters of the surrogate model using
both pre-trained and random parameters. Subsequently, we save the models in the
intermediate training process to obtain a diverse set of surrogate models.
During the HMA stage, we enhance the feature maps of the diversified surrogate
models by incorporating beneficial perturbations, thereby further improving the
transferability. Experimental results demonstrate that our proposed attack
method can effectively enhance the transferability of the crafted adversarial
face examples.


http://arxiv.org/abs/2411.15673
Semantic Shield: Defending Vision-Language Models Against Backdooring and Poisoning via Fine-grained Knowledge Alignment. (4%)
Alvi Md Ishmam; Christopher Thomas
  In recent years there has been enormous interest in vision-language models
trained using self-supervised objectives. However, the use of large-scale
datasets scraped from the web for training also makes these models vulnerable
to potential security threats, such as backdooring and poisoning attacks. In
this paper, we propose a method for mitigating such attacks on contrastively
trained vision-language models. Our approach leverages external knowledge
extracted from a language model to prevent models from learning correlations
between image regions which lack strong alignment with external knowledge. We
do this by imposing constraints to enforce that attention paid by the model to
visual regions is proportional to the alignment of those regions with external
knowledge. We conduct extensive experiments using a variety of recent
backdooring and poisoning attacks on multiple datasets and architectures. Our
results clearly demonstrate that our proposed approach is highly effective at
defending against such attacks across multiple settings, while maintaining
model utility and without requiring any changes at inference time


http://arxiv.org/abs/2411.16746
LoBAM: LoRA-Based Backdoor Attack on Model Merging. (2%)
Ming Yin; Jingyang Zhang; Jingwei Sun; Minghong Fang; Hai Li; Yiran Chen
  Model merging is an emerging technique that integrates multiple models
fine-tuned on different tasks to create a versatile model that excels in
multiple domains. This scheme, in the meantime, may open up backdoor attack
opportunities where one single malicious model can jeopardize the integrity of
the merged model. Existing works try to demonstrate the risk of such attacks by
assuming substantial computational resources, focusing on cases where the
attacker can fully fine-tune the pre-trained model. Such an assumption,
however, may not be feasible given the increasing size of machine learning
models. In practice where resources are limited and the attacker can only
employ techniques like Low-Rank Adaptation (LoRA) to produce the malicious
model, it remains unclear whether the attack can still work and pose threats.
In this work, we first identify that the attack efficacy is significantly
diminished when using LoRA for fine-tuning. Then, we propose LoBAM, a method
that yields high attack success rate with minimal training resources. The key
idea of LoBAM is to amplify the malicious weights in an intelligent way that
effectively enhances the attack efficacy. We demonstrate that our design can
lead to improved attack success rate through both theoretical proof and
extensive empirical experiments across various model merging scenarios.
Moreover, we show that our method has strong stealthiness and is difficult to
detect.


http://arxiv.org/abs/2411.15246
Exploring the Robustness and Transferability of Patch-Based Adversarial Attacks in Quantized Neural Networks. (99%)
Amira Guesmi; Bassem Ouni; Muhammad Shafique
  Quantized neural networks (QNNs) are increasingly used for efficient
deployment of deep learning models on resource-constrained platforms, such as
mobile devices and edge computing systems. While quantization reduces model
size and computational demands, its impact on adversarial robustness-especially
against patch-based attacks-remains inadequately addressed. Patch-based
attacks, characterized by localized, high-visibility perturbations, pose
significant security risks due to their transferability and resilience. In this
study, we systematically evaluate the vulnerability of QNNs to patch-based
adversarial attacks across various quantization levels and architectures,
focusing on factors that contribute to the robustness of these attacks. Through
experiments analyzing feature representations, quantization strength, gradient
alignment, and spatial sensitivity, we find that patch attacks consistently
achieve high success rates across bitwidths and architectures, demonstrating
significant transferability even in heavily quantized models. Contrary to the
expectation that quantization might enhance adversarial defenses, our results
show that QNNs remain highly susceptible to patch attacks due to the
persistence of distinct, localized features within quantized representations.
These findings underscore the need for quantization-aware defenses that address
the specific challenges posed by patch-based attacks. Our work contributes to a
deeper understanding of adversarial robustness in QNNs and aims to guide future
research in developing secure, quantization-compatible defenses for real-world
applications.


http://arxiv.org/abs/2411.14834
Gradient Masking All-at-Once: Ensemble Everything Everywhere Is Not Robust. (99%)
Jie Zhang; Kristina Nikolić; Nicholas Carlini; Florian Tramèr
  Ensemble everything everywhere is a defense to adversarial examples that was
recently proposed to make image classifiers robust. This defense works by
ensembling a model's intermediate representations at multiple noisy image
resolutions, producing a single robust classification. This defense was shown
to be effective against multiple state-of-the-art attacks. Perhaps even more
convincingly, it was shown that the model's gradients are perceptually aligned:
attacks against the model produce noise that perceptually resembles the
targeted class.
  In this short note, we show that this defense is not robust to adversarial
attack. We first show that the defense's randomness and ensembling method cause
severe gradient masking. We then use standard adaptive attack techniques to
reduce the defense's robust accuracy from 48% to 1% on CIFAR-100 and from 62%
to 4% on CIFAR-10, under the $\ell_\infty$-norm threat model with
$\varepsilon=8/255$.


http://arxiv.org/abs/2411.16721
Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreaks. (98%)
Han Wang; Gang Wang; Huan Zhang
  Vision Language Models (VLMs) can produce unintended and harmful content when
exposed to adversarial attacks, particularly because their vision capabilities
create new vulnerabilities. Existing defenses, such as input preprocessing,
adversarial training, and response evaluation-based methods, are often
impractical for real-world deployment due to their high costs. To address this
challenge, we propose ASTRA, an efficient and effective defense by adaptively
steering models away from adversarial feature directions to resist VLM attacks.
Our key procedures involve finding transferable steering vectors representing
the direction of harmful response and applying adaptive activation steering to
remove these directions at inference time. To create effective steering
vectors, we randomly ablate the visual tokens from the adversarial images and
identify those most strongly associated with jailbreaks. These tokens are then
used to construct steering vectors. During inference, we perform the adaptive
steering method that involves the projection between the steering vectors and
calibrated activation, resulting in little performance drops on benign inputs
while strongly avoiding harmful outputs under adversarial inputs. Extensive
experiments across multiple models and baselines demonstrate our
state-of-the-art performance and high efficiency in mitigating jailbreak risks.
Additionally, ASTRA exhibits good transferability, defending against both
unseen attacks at design time (i.e., structured-based attacks) and adversarial
images from diverse distributions.


http://arxiv.org/abs/2411.15265
Derivative-Free Diffusion Manifold-Constrained Gradient for Unified XAI. (45%)
Won Jun Kim; Hyungjin Chung; Jaemin Kim; Sangmin Lee; Byeongsu Sim; Jong Chul Ye
  Gradient-based methods are a prototypical family of explainability
techniques, especially for image-based models. Nonetheless, they have several
shortcomings in that they (1) require white-box access to models, (2) are
vulnerable to adversarial attacks, and (3) produce attributions that lie off
the image manifold, leading to explanations that are not actually faithful to
the model and do not align well with human perception. To overcome these
challenges, we introduce Derivative-Free Diffusion Manifold-Constrainted
Gradients (FreeMCG), a novel method that serves as an improved basis for
explainability of a given neural network than the traditional gradient.
Specifically, by leveraging ensemble Kalman filters and diffusion models, we
derive a derivative-free approximation of the model's gradient projected onto
the data manifold, requiring access only to the model's outputs. We demonstrate
the effectiveness of FreeMCG by applying it to both counterfactual generation
and feature attribution, which have traditionally been treated as distinct
tasks. Through comprehensive evaluation on both tasks, counterfactual
explanation and feature attribution, we show that our method yields
state-of-the-art results while preserving the essential properties expected of
XAI tools.


http://arxiv.org/abs/2411.14842
Who Can Withstand Chat-Audio Attacks? An Evaluation Benchmark for Large Language Models. (41%)
Wanqi Yang; Yanda Li; Meng Fang; Yunchao Wei; Tianyi Zhou; Ling Chen
  Adversarial audio attacks pose a significant threat to the growing use of
large language models (LLMs) in voice-based human-machine interactions. While
existing research has primarily focused on model-specific adversarial methods,
real-world applications demand a more generalizable and universal approach to
audio adversarial attacks. In this paper, we introduce the Chat-Audio Attacks
(CAA) benchmark including four distinct types of audio attacks, which aims to
explore the the vulnerabilities of LLMs to these audio attacks in
conversational scenarios. To evaluate the robustness of LLMs, we propose three
evaluation strategies: Standard Evaluation, utilizing traditional metrics to
quantify model performance under attacks; GPT-4o-Based Evaluation, which
simulates real-world conversational complexities; and Human Evaluation,
offering insights into user perception and trust. We evaluate six
state-of-the-art LLMs with voice interaction capabilities, including
Gemini-1.5-Pro, GPT-4o, and others, using three distinct evaluation methods on
the CAA benchmark. Our comprehensive analysis reveals the impact of four types
of audio attacks on the performance of these models, demonstrating that GPT-4o
exhibits the highest level of resilience.


http://arxiv.org/abs/2411.14738
Universal and Context-Independent Triggers for Precise Control of LLM Outputs. (31%)
Jiashuo Liang; Guancheng Li; Yang Yu
  Large language models (LLMs) have been widely adopted in applications such as
automated content generation and even critical decision-making systems.
However, the risk of prompt injection allows for potential manipulation of LLM
outputs. While numerous attack methods have been documented, achieving full
control over these outputs remains challenging, often requiring experienced
attackers to make multiple attempts and depending heavily on the prompt
context. Recent advancements in gradient-based white-box attack techniques have
shown promise in tasks like jailbreaks and system prompt leaks. Our research
generalizes gradient-based attacks to find a trigger that is (1) Universal:
effective irrespective of the target output; (2) Context-Independent: robust
across diverse prompt contexts; and (3) Precise Output: capable of manipulating
LLM inputs to yield any specified output with high accuracy. We propose a novel
method to efficiently discover such triggers and assess the effectiveness of
the proposed attack. Furthermore, we discuss the substantial threats posed by
such attacks to LLM-based applications, highlighting the potential for
adversaries to taking over the decisions and actions made by AI agents.


http://arxiv.org/abs/2411.14865
Benchmarking the Robustness of Optical Flow Estimation to Corruptions. (13%)
Zhonghua Yi; Hao Shi; Qi Jiang; Yao Gao; Ze Wang; Yufan Zhang; Kailun Yang; Kaiwei Wang
  Optical flow estimation is extensively used in autonomous driving and video
editing. While existing models demonstrate state-of-the-art performance across
various benchmarks, the robustness of these methods has been infrequently
investigated. Despite some research focusing on the robustness of optical flow
models against adversarial attacks, there has been a lack of studies
investigating their robustness to common corruptions. Taking into account the
unique temporal characteristics of optical flow, we introduce 7 temporal
corruptions specifically designed for benchmarking the robustness of optical
flow models, in addition to 17 classical single-image corruptions, in which
advanced PSF Blur simulation method is performed. Two robustness benchmarks,
KITTI-FC and GoPro-FC, are subsequently established as the first corruption
robustness benchmark for optical flow estimation, with Out-Of-Domain (OOD) and
In-Domain (ID) settings to facilitate comprehensive studies. Robustness
metrics, Corruption Robustness Error (CRE), Corruption Robustness Error ratio
(CREr), and Relative Corruption Robustness Error (RCRE) are further introduced
to quantify the optical flow estimation robustness. 29 model variants from 15
optical flow methods are evaluated, yielding 10 intriguing observations, such
as 1) the absolute robustness of the model is heavily dependent on the
estimation performance; 2) the corruptions that diminish local information are
more serious than that reduce visual effects. We also give suggestions for the
design and application of optical flow models. We anticipate that our benchmark
will serve as a foundational resource for advancing research in robust optical
flow estimation. The benchmarks and source code will be released at
https://github.com/ZhonghuaYi/optical_flow_robustness_benchmark.


http://arxiv.org/abs/2411.15439
Twin Trigger Generative Networks for Backdoor Attacks against Object Detection. (4%)
Zhiying Li; Zhi Liu; Guanggang Geng; Shreyank N Gowda; Shuyuan Lin; Jian Weng; Xiaobo Jin
  Object detectors, which are widely used in real-world applications, are
vulnerable to backdoor attacks. This vulnerability arises because many users
rely on datasets or pre-trained models provided by third parties due to
constraints on data and resources. However, most research on backdoor attacks
has focused on image classification, with limited investigation into object
detection. Furthermore, the triggers for most existing backdoor attacks on
object detection are manually generated, requiring prior knowledge and
consistent patterns between the training and inference stages. This approach
makes the attacks either easy to detect or difficult to adapt to various
scenarios. To address these limitations, we propose novel twin trigger
generative networks in the frequency domain to generate invisible triggers for
implanting stealthy backdoors into models during training, and visible triggers
for steady activation during inference, making the attack process difficult to
trace. Specifically, for the invisible trigger generative network, we deploy a
Gaussian smoothing layer and a high-frequency artifact classifier to enhance
the stealthiness of backdoor implantation in object detectors. For the visible
trigger generative network, we design a novel alignment loss to optimize the
visible triggers so that they differ from the original patterns but still align
with the malicious activation behavior of the invisible triggers. Extensive
experimental results and analyses prove the possibility of using different
triggers in the training stage and the inference stage, and demonstrate the
attack effectiveness of our proposed visible trigger and invisible trigger
generative networks, significantly reducing the mAP_0.5 of the object detectors
by 70.0% and 84.5%, including YOLOv5 and YOLOv7 with different settings,
respectively.


http://arxiv.org/abs/2411.14937
Geminio: Language-Guided Gradient Inversion Attacks in Federated Learning. (2%)
Junjie Shan; Ziqi Zhao; Jialin Lu; Rui Zhang; Siu Ming Yiu; Ka-Ho Chow
  Foundation models that bridge vision and language have made significant
progress, inspiring numerous life-enriching applications. However, their
potential for misuse to introduce new threats remains largely unexplored. This
paper reveals that vision-language models (VLMs) can be exploited to overcome
longstanding limitations in gradient inversion attacks (GIAs) within federated
learning (FL), where an FL server reconstructs private data samples from
gradients shared by victim clients. Current GIAs face challenges in
reconstructing high-resolution images, especially when the victim has a large
local data batch. While focusing reconstruction on valuable samples rather than
the entire batch is promising, existing methods lack the flexibility to allow
attackers to specify their target data. In this paper, we introduce Geminio,
the first approach to transform GIAs into semantically meaningful, targeted
attacks. Geminio enables a brand new privacy attack experience: attackers can
describe, in natural language, the types of data they consider valuable, and
Geminio will prioritize reconstruction to focus on those high-value samples.
This is achieved by leveraging a pretrained VLM to guide the optimization of a
malicious global model that, when shared with and optimized by a victim,
retains only gradients of samples that match the attacker-specified query.
Extensive experiments demonstrate Geminio's effectiveness in pinpointing and
reconstructing targeted samples, with high success rates across complex
datasets under FL and large batch sizes and showing resilience against existing
defenses.


http://arxiv.org/abs/2411.15306
Heavy-tailed Contamination is Easier than Adversarial Contamination. (1%)
Yeshwanth Cherapanamjeri; Daniel Lee
  A large body of work in the statistics and computer science communities
dating back to Huber (Huber, 1960) has led to statistically and computationally
efficient outlier-robust estimators. Two particular outlier models have
received significant attention: the adversarial and heavy-tailed models. While
the former models outliers as the result of a malicious adversary manipulating
the data, the latter relaxes distributional assumptions on the data allowing
outliers to naturally occur as part of the data generating process. In the
first setting, the goal is to develop estimators robust to the largest fraction
of outliers while in the second, one seeks estimators to combat the loss of
statistical efficiency, where the dependence on the failure probability is
paramount.
  Despite these distinct motivations, the algorithmic approaches to both these
settings have converged, prompting questions on the relationship between the
models. In this paper, we investigate and provide a principled explanation for
this phenomenon. First, we prove that any adversarially robust estimator is
also resilient to heavy-tailed outliers for any statistical estimation problem
with i.i.d data. As a corollary, optimal adversarially robust estimators for
mean estimation, linear regression, and covariance estimation are also optimal
heavy-tailed estimators. Conversely, for arguably the simplest high-dimensional
estimation task of mean estimation, we construct heavy-tailed estimators whose
application to the adversarial setting requires any black-box reduction to
remove almost all the outliers in the data. Taken together, our results imply
that heavy-tailed estimation is likely easier than adversarially robust
estimation opening the door to novel algorithmic approaches for the
heavy-tailed setting. Additionally, confidence intervals obtained for
adversarially robust estimation also hold with high-probability.


http://arxiv.org/abs/2411.15367
Exploiting Watermark-Based Defense Mechanisms in Text-to-Image Diffusion Models for Unauthorized Data Usage. (1%)
Soumil Datta; Shih-Chieh Dai; Leo Yu; Guanhong Tao
  Text-to-image diffusion models, such as Stable Diffusion, have shown
exceptional potential in generating high-quality images. However, recent
studies highlight concerns over the use of unauthorized data in training these
models, which may lead to intellectual property infringement or privacy
violations. A promising approach to mitigate these issues is to apply a
watermark to images and subsequently check if generative models reproduce
similar watermark features. In this paper, we examine the robustness of various
watermark-based protection methods applied to text-to-image models. We observe
that common image transformations are ineffective at removing the watermark
effect. Therefore, we propose RATTAN, that leverages the diffusion process to
conduct controlled image generation on the protected input, preserving the
high-level features of the input while ignoring the low-level details utilized
by watermarks. A small number of generated images are then used to fine-tune
protected models. Our experiments on three datasets and 140 text-to-image
diffusion models reveal that existing state-of-the-art protections are not
robust against RATTAN.


http://arxiv.org/abs/2411.14946
Reliable Evaluation of Attribution Maps in CNNs: A Perturbation-Based Approach. (1%)
Lars Nieradzik; Henrike Stephani; Janis Keuper
  In this paper, we present an approach for evaluating attribution maps, which
play a central role in interpreting the predictions of convolutional neural
networks (CNNs). We show that the widely used insertion/deletion metrics are
susceptible to distribution shifts that affect the reliability of the ranking.
Our method proposes to replace pixel modifications with adversarial
perturbations, which provides a more robust evaluation framework. By using
smoothness and monotonicity measures, we illustrate the effectiveness of our
approach in correcting distribution shifts. In addition, we conduct the most
comprehensive quantitative and qualitative assessment of attribution maps to
date. Introducing baseline attribution maps as sanity checks, we find that our
metric is the only contender to pass all checks. Using Kendall's $\tau$ rank
correlation coefficient, we show the increased consistency of our metric across
15 dataset-architecture combinations. Of the 16 attribution maps tested, our
results clearly show SmoothGrad to be the best map currently available. This
research makes an important contribution to the development of attribution maps
by providing a reliable and consistent evaluation framework. To ensure
reproducibility, we will provide the code along with our results.


http://arxiv.org/abs/2411.14263
Generating Realistic Adversarial Examples for Business Processes using Variational Autoencoders. (99%)
Alexander Stevens; Jari Peeperkorn; Smedt Johannes De; Weerdt Jochen De
  In predictive process monitoring, predictive models are vulnerable to
adversarial attacks, where input perturbations can lead to incorrect
predictions. Unlike in computer vision, where these perturbations are designed
to be imperceptible to the human eye, the generation of adversarial examples in
predictive process monitoring poses unique challenges. Minor changes to the
activity sequences can create improbable or even impossible scenarios to occur
due to underlying constraints such as regulatory rules or process constraints.
To address this, we focus on generating realistic adversarial examples tailored
to the business process context, in contrast to the imperceptible, pixel-level
changes commonly seen in computer vision adversarial attacks. This paper
introduces two novel latent space attacks, which generate adversaries by adding
noise to the latent space representation of the input data, rather than
directly modifying the input attributes. These latent space methods are
domain-agnostic and do not rely on process-specific knowledge, as we restrict
the generation of adversarial examples to the learned class-specific data
distributions by directly perturbing the latent space representation of the
business process executions. We evaluate these two latent space methods with
six other adversarial attacking methods on eleven real-life event logs and four
predictive models. The first three attacking methods directly permute the
activities of the historically observed business process executions. The fourth
method constrains the adversarial examples to lie within the same data
distribution as the original instances, by projecting the adversarial examples
to the original data distribution.


http://arxiv.org/abs/2411.14424
Learning Fair Robustness via Domain Mixup. (81%)
Meiyu Zhong; Ravi Tandon
  Adversarial training is one of the predominant techniques for training
classifiers that are robust to adversarial attacks. Recent work, however has
found that adversarial training, which makes the overall classifier robust, it
does not necessarily provide equal amount of robustness for all classes. In
this paper, we propose the use of mixup for the problem of learning fair robust
classifiers, which can provide similar robustness across all classes.
Specifically, the idea is to mix inputs from the same classes and perform
adversarial training on mixed up inputs. We present a theoretical analysis of
this idea for the case of linear classifiers and show that mixup combined with
adversarial training can provably reduce the class-wise robustness disparity.
This method not only contributes to reducing the disparity in class-wise
adversarial risk, but also the class-wise natural risk. Complementing our
theoretical analysis, we also provide experimental results on both synthetic
data and the real world dataset (CIFAR-10), which shows improvement in class
wise disparities for both natural and adversarial risks.


http://arxiv.org/abs/2411.14133
GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMs. (78%)
Advik Raj Basani; Xiao Zhang
  Large Language Models (LLMs) have shown impressive proficiency across a range
of natural language processing tasks yet remain vulnerable to adversarial
prompts, known as jailbreak attacks, carefully designed to elicit harmful
responses from LLMs. Traditional methods rely on manual heuristics, which
suffer from limited generalizability. While being automatic, optimization-based
attacks often produce unnatural jailbreak prompts that are easy to detect by
safety filters or require high computational overhead due to discrete token
optimization. Witnessing the limitations of existing jailbreak methods, we
introduce Generative Adversarial Suffix Prompter (GASP), a novel framework that
combines human-readable prompt generation with Latent Bayesian Optimization
(LBO) to improve adversarial suffix creation in a fully black-box setting. GASP
leverages LBO to craft adversarial suffixes by efficiently exploring continuous
embedding spaces, gradually optimizing the model to improve attack efficacy
while balancing prompt coherence through a targeted iterative refinement
procedure. Our experiments show that GASP can generate natural jailbreak
prompts, significantly improving attack success rates, reducing training times,
and accelerating inference speed, thus making it an efficient and scalable
solution for red-teaming LLMs.


http://arxiv.org/abs/2411.15244
Adversarial Prompt Distillation for Vision-Language Models. (75%)
Lin Luo; Xin Wang; Bojia Zi; Shihao Zhao; Xingjun Ma
  Large pre-trained Vision-Language Models (VLMs) such as Contrastive
Language-Image Pre-Training (CLIP) have been shown to be susceptible to
adversarial attacks, raising concerns about their deployment in safety-critical
scenarios like autonomous driving and medical diagnosis. One promising approach
for improving the robustness of pre-trained VLMs is Adversarial Prompt Tuning
(APT), which combines adversarial training with prompt tuning. However,
existing APT methods are mostly single-modal methods that design prompt(s) for
only the visual or textual modality, limiting their effectiveness in either
robustness or clean accuracy. In this work, we propose a novel method called
Adversarial Prompt Distillation (APD) that combines APT with knowledge
distillation to boost the adversarial robustness of CLIP. Specifically, APD is
a bimodal method that adds prompts for both the visual and textual modalities
while leveraging a cleanly pre-trained teacher CLIP model to distill and boost
the performance of the student CLIP model on downstream tasks. Extensive
experiments on multiple benchmark datasets demonstrate the superiority of our
APD over the current state-of-the-art APT methods in terms of both natural and
adversarial performances. The effectiveness of our APD method validates the
possibility of using a non-robust teacher to improve the generalization and
robustness of VLMs.


http://arxiv.org/abs/2411.14243
AnywhereDoor: Multi-Target Backdoor Attacks on Object Detection. (74%)
Jialin Lu; Junjie Shan; Ziqi Zhao; Ka-Ho Chow
  As object detection becomes integral to many safety-critical applications,
understanding its vulnerabilities is essential. Backdoor attacks, in
particular, pose a significant threat by implanting hidden backdoor in a victim
model, which adversaries can later exploit to trigger malicious behaviors
during inference. However, current backdoor techniques are limited to static
scenarios where attackers must define a malicious objective before training,
locking the attack into a predetermined action without inference-time
adaptability. Given the expressive output space in object detection, including
object existence detection, bounding box estimation, and object classification,
the feasibility of implanting a backdoor that provides inference-time control
with a high degree of freedom remains unexplored. This paper introduces
AnywhereDoor, a flexible backdoor attack tailored for object detection. Once
implanted, AnywhereDoor enables adversaries to specify different attack types
(object vanishing, fabrication, or misclassification) and configurations
(untargeted or targeted with specific classes) to dynamically control detection
behavior. This flexibility is achieved through three key innovations: (i)
objective disentanglement to support a broader range of attack combinations
well beyond what existing methods allow; (ii) trigger mosaicking to ensure
backdoor activations are robust, even against those object detectors that
extract localized regions from the input image for recognition; and (iii)
strategic batching to address object-level data imbalances that otherwise
hinders a balanced manipulation. Extensive experiments demonstrate that
AnywhereDoor provides attackers with a high degree of control, achieving an
attack success rate improvement of nearly 80% compared to adaptations of
existing methods for such flexible control.


http://arxiv.org/abs/2411.14351
Indiscriminate Disruption of Conditional Inference on Multivariate Gaussians. (50%)
William N. Caballero; Matthew LaRosa; Alexander Fisher; Vahid Tarokh
  The multivariate Gaussian distribution underpins myriad operations-research,
decision-analytic, and machine-learning models (e.g., Bayesian optimization,
Gaussian influence diagrams, and variational autoencoders). However, despite
recent advances in adversarial machine learning (AML), inference for Gaussian
models in the presence of an adversary is notably understudied. Therefore, we
consider a self-interested attacker who wishes to disrupt a decisionmaker's
conditional inference and subsequent actions by corrupting a set of evidentiary
variables. To avoid detection, the attacker also desires the attack to appear
plausible wherein plausibility is determined by the density of the corrupted
evidence. We consider white- and grey-box settings such that the attacker has
complete and incomplete knowledge about the decisionmaker's underlying
multivariate Gaussian distribution, respectively. Select instances are shown to
reduce to quadratic and stochastic quadratic programs, and structural
properties are derived to inform solution methods. We assess the impact and
efficacy of these attacks in three examples, including, real estate evaluation,
interest rate estimation and signals processing. Each example leverages an
alternative underlying model, thereby highlighting the attacks' broad
applicability. Through these applications, we also juxtapose the behavior of
the white- and grey-box attacks to understand how uncertainty and structure
affect attacker behavior.


http://arxiv.org/abs/2411.14718
GraphTheft: Quantifying Privacy Risks in Graph Prompt Learning. (4%)
Jiani Zhu; Xi Lin; Yuxin Qi; Qinghua Mao
  Graph Prompt Learning (GPL) represents an innovative approach in graph
representation learning, enabling task-specific adaptations by fine-tuning
prompts without altering the underlying pre-trained model. Despite its growing
prominence, the privacy risks inherent in GPL remain unexplored. In this study,
we provide the first evaluation of privacy leakage in GPL across three attacker
capabilities: black-box attacks when GPL as a service, and scenarios where node
embeddings and prompt representations are accessible to third parties. We
assess GPL's privacy vulnerabilities through Attribute Inference Attacks (AIAs)
and Link Inference Attacks (LIAs), finding that under any capability, attackers
can effectively infer the properties and relationships of sensitive nodes, and
the success rate of inference on some data sets is as high as 98%. Importantly,
while targeted inference attacks on specific prompts (e.g., GPF-plus) maintain
high success rates, our analysis suggests that the prompt-tuning in GPL does
not significantly elevate privacy risks compared to traditional GNNs. To
mitigate these risks, we explored defense mechanisms, identifying that
Laplacian noise perturbation can substantially reduce inference success, though
balancing privacy protection with model performance remains challenging. This
work highlights critical privacy risks in GPL, offering new insights and
foundational directions for future privacy-preserving strategies in graph
learning.


http://arxiv.org/abs/2411.14502
Global Challenge for Safe and Secure LLMs Track 1. (4%)
Xiaojun Jia; Yihao Huang; Yang Liu; Peng Yan Tan; Weng Kuan Yau; Mun-Thye Mak; Xin Ming Sim; Wee Siong Ng; See Kiong Ng; Hanqing Liu; Lifeng Zhou; Huanqian Yan; Xiaobing Sun; Wei Liu; Long Wang; Yiming Qian; Yong Liu; Junxiao Yang; Zhexin Zhang; Leqi Lei; Renmiao Chen; Yida Lu; Shiyao Cui; Zizhou Wang; Shaohua Li; Yan Wang; Rick Siow Mong Goh; Liangli Zhen; Yingjie Zhang; Zhe Zhao
  This paper introduces the Global Challenge for Safe and Secure Large Language
Models (LLMs), a pioneering initiative organized by AI Singapore (AISG) and the
CyberSG R&D Programme Office (CRPO) to foster the development of advanced
defense mechanisms against automated jailbreaking attacks. With the increasing
integration of LLMs in critical sectors such as healthcare, finance, and public
administration, ensuring these models are resilient to adversarial attacks is
vital for preventing misuse and upholding ethical standards. This competition
focused on two distinct tracks designed to evaluate and enhance the robustness
of LLM security frameworks. Track 1 tasked participants with developing
automated methods to probe LLM vulnerabilities by eliciting undesirable
responses, effectively testing the limits of existing safety protocols within
LLMs. Participants were challenged to devise techniques that could bypass
content safeguards across a diverse array of scenarios, from offensive language
to misinformation and illegal activities. Through this process, Track 1 aimed
to deepen the understanding of LLM vulnerabilities and provide insights for
creating more resilient models.


http://arxiv.org/abs/2411.14681
TrojanEdit: Backdooring Text-Based Image Editing Models. (3%)
Ji Guo; Peihong Chen; Wenbo Jiang; Guoming Lu
  As diffusion models have achieved success in image generation tasks, many
studies have extended them to other related fields like image editing. Unlike
image generation, image editing aims to modify an image based on user requests
while keeping other parts of the image unchanged. Among these, text-based image
editing is the most representative task.Some studies have shown that diffusion
models are vulnerable to backdoor attacks, where attackers may poison the
training data to inject the backdoor into models. However, previous backdoor
attacks on diffusion models primarily focus on image generation models without
considering image editing models. Given that image editing models accept
multimodal inputs, it raises a new question regarding the effectiveness of
different modalities triggers in backdoor attacks on these models. To address
this question, we propose a backdoor attack framework for image editing models,
named TrojanEdit, which can handle different modalities triggers. We explore
five types of visual triggers, three types of textual triggers, and combine
them together as fifteen types of multimodal triggers, conducting extensive
experiments for three types of backdoor attack goals. Our experimental results
show that the image editing model has a backdoor bias for texture triggers.
Compared to visual triggers, textual triggers have stronger attack
effectiveness but also cause more damage to the model's normal functionality.
Furthermore, we found that multimodal triggers can achieve a good balance
between the attack effectiveness and model's normal functionality.


http://arxiv.org/abs/2411.14215
Evaluating the Robustness of Analogical Reasoning in Large Language Models. (1%)
Martha Lewis; Melanie Mitchell
  LLMs have performed well on several reasoning benchmarks, including ones that
test analogical reasoning abilities. However, there is debate on the extent to
which they are performing general abstract reasoning versus employing
non-robust processes, e.g., that overly rely on similarity to pre-training
data. Here we investigate the robustness of analogy-making abilities previously
claimed for LLMs on three of four domains studied by Webb, Holyoak, and Lu
(2023): letter-string analogies, digit matrices, and story analogies. For each
domain we test humans and GPT models on robustness to variants of the original
analogy problems that test the same abstract reasoning abilities but are likely
dissimilar from tasks in the pre-training data. The performance of a system
that uses robust abstract reasoning should not decline substantially on these
variants.
  On simple letter-string analogies, we find that while the performance of
humans remains high for two types of variants we tested, the GPT models'
performance declines sharply. This pattern is less pronounced as the complexity
of these problems is increased, as both humans and GPT models perform poorly on
both the original and variant problems requiring more complex analogies. On
digit-matrix problems, we find a similar pattern but only on one out of the two
types of variants we tested. On story-based analogy problems, we find that,
unlike humans, the performance of GPT models are susceptible to answer-order
effects, and that GPT models also may be more sensitive than humans to
paraphrasing.
  This work provides evidence that LLMs often lack the robustness of zero-shot
human analogy-making, exhibiting brittleness on most of the variations we
tested. More generally, this work points to the importance of carefully
evaluating AI systems not only for accuracy but also robustness when testing
their cognitive capabilities.


http://arxiv.org/abs/2411.14516
Memory Backdoor Attacks on Neural Networks. (1%)
Eden Luzon; Guy Amit; Roy Weiss; Yisroel Mirsky
  Neural networks, such as image classifiers, are frequently trained on
proprietary and confidential datasets. It is generally assumed that once
deployed, the training data remains secure, as adversaries are limited to query
response interactions with the model, where at best, fragments of arbitrary
data can be inferred without any guarantees on their authenticity. In this
paper, we propose the memory backdoor attack, where a model is covertly trained
to memorize specific training samples and later selectively output them when
triggered with an index pattern. What makes this attack unique is that it (1)
works even when the tasks conflict (making a classifier output images), (2)
enables the systematic extraction of training samples from deployed models and
(3) offers guarantees on the extracted authenticity of the data. We demonstrate
the attack on image classifiers, segmentation models, and a large language
model (LLM). We demonstrate the attack on image classifiers, segmentation
models, and a large language model (LLM). With this attack, it is possible to
hide thousands of images and texts in modern vision architectures and LLMs
respectively, all while maintaining model performance. The memory back door
attack poses a significant threat not only to conventional model deployments
but also to federated learning paradigms and other modern frameworks.
Therefore, we suggest an efficient and effective countermeasure that can be
immediately applied and advocate for further work on the topic.


http://arxiv.org/abs/2411.15210
Towards Million-Scale Adversarial Robustness Evaluation With Stronger Individual Attacks. (98%)
Yong Xie; Weijie Zheng; Hanxun Huang; Guangnan Ye; Xingjun Ma
  As deep learning models are increasingly deployed in safety-critical
applications, evaluating their vulnerabilities to adversarial perturbations is
essential for ensuring their reliability and trustworthiness. Over the past
decade, a large number of white-box adversarial robustness evaluation methods
(i.e., attacks) have been proposed, ranging from single-step to multi-step
methods and from individual to ensemble methods. Despite these advances,
challenges remain in conducting meaningful and comprehensive robustness
evaluations, particularly when it comes to large-scale testing and ensuring
evaluations reflect real-world adversarial risks. In this work, we focus on
image classification models and propose a novel individual attack method,
Probability Margin Attack (PMA), which defines the adversarial margin in the
probability space rather than the logits space. We analyze the relationship
between PMA and existing cross-entropy or logits-margin-based attacks, and show
that PMA can outperform the current state-of-the-art individual methods.
Building on PMA, we propose two types of ensemble attacks that balance
effectiveness and efficiency. Furthermore, we create a million-scale dataset,
CC1M, derived from the existing CC3M dataset, and use it to conduct the first
million-scale white-box adversarial robustness evaluation of
adversarially-trained ImageNet models. Our findings provide valuable insights
into the robustness gaps between individual versus ensemble attacks and
small-scale versus million-scale evaluations.


http://arxiv.org/abs/2411.13136
TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models. (96%)
Xin Wang; Kai Chen; Jiaming Zhang; Jingjing Chen; Xingjun Ma
  Large pre-trained Vision-Language Models (VLMs) such as CLIP have
demonstrated excellent zero-shot generalizability across various downstream
tasks. However, recent studies have shown that the inference performance of
CLIP can be greatly degraded by small adversarial perturbations, especially its
visual modality, posing significant safety threats. To mitigate this
vulnerability, in this paper, we propose a novel defense method called
Test-Time Adversarial Prompt Tuning (TAPT) to enhance the inference robustness
of CLIP against visual adversarial attacks. TAPT is a test-time defense method
that learns defensive bimodal (textual and visual) prompts to robustify the
inference process of CLIP. Specifically, it is an unsupervised method that
optimizes the defensive prompts for each test sample by minimizing a multi-view
entropy and aligning adversarial-clean distributions. We evaluate the
effectiveness of TAPT on 11 benchmark datasets, including ImageNet and 10 other
zero-shot datasets, demonstrating that it enhances the zero-shot adversarial
robustness of the original CLIP by at least 48.9% against AutoAttack (AA),
while largely maintaining performance on clean examples. Moreover, TAPT
outperforms existing adversarial prompt tuning methods across various
backbones, achieving an average robustness improvement of at least 36.6%.


http://arxiv.org/abs/2411.13116
Provably Efficient Action-Manipulation Attack Against Continuous Reinforcement Learning. (86%)
Zhi Luo; Xiyuan Yang; Pan Zhou; Di Wang
  Manipulating the interaction trajectories between the intelligent agent and
the environment can control the agent's training and behavior, exposing the
potential vulnerabilities of reinforcement learning (RL). For example, in
Cyber-Physical Systems (CPS) controlled by RL, the attacker can manipulate the
actions of the adopted RL to other actions during the training phase, which
will lead to bad consequences. Existing work has studied action-manipulation
attacks in tabular settings, where the states and actions are discrete. As seen
in many up-and-coming RL applications, such as autonomous driving, continuous
action space is widely accepted, however, its action-manipulation attacks have
not been thoroughly investigated yet. In this paper, we consider this crucial
problem in both white-box and black-box scenarios. Specifically, utilizing the
knowledge derived exclusively from trajectories, we propose a black-box attack
algorithm named LCBT, which uses the Monte Carlo tree search method for
efficient action searching and manipulation. Additionally, we demonstrate that
for an agent whose dynamic regret is sub-linearly related to the total number
of steps, LCBT can teach the agent to converge to target policies with only
sublinear attack cost, i.e., $O\left(\mathcal{R}(T) + MH^3K^E\log
(MT)\right)(0<E<1)$, where $H$ is the number of steps per episode, $K$ is the
total number of episodes, $T=KH$ is the total number of steps, $M$ is the
number of subspaces divided in the state space, and $\mathcal{R}(T)$ is the
bound of the RL algorithm's regret. We conduct our proposed attack methods on
three aggressive algorithms: DDPG, PPO, and TD3 in continuous settings, which
show a promising attack performance.


http://arxiv.org/abs/2411.13778
A Survey on Adversarial Robustness of LiDAR-based Machine Learning Perception in Autonomous Vehicles. (86%)
Junae Kim; Amardeep Kaur
  In autonomous driving, the combination of AI and vehicular technology offers
great potential. However, this amalgamation comes with vulnerabilities to
adversarial attacks. This survey focuses on the intersection of Adversarial
Machine Learning (AML) and autonomous systems, with a specific focus on
LiDAR-based systems. We comprehensively explore the threat landscape,
encompassing cyber-attacks on sensors and adversarial perturbations.
Additionally, we investigate defensive strategies employed in countering these
threats. This paper endeavors to present a concise overview of the challenges
and advances in securing autonomous driving systems against adversarial
threats, emphasizing the need for robust defenses to ensure safety and
security.


http://arxiv.org/abs/2411.15222
Rethinking the Intermediate Features in Adversarial Attacks: Misleading Robotic Models via Adversarial Distillation. (68%)
Ke Wuhan University Zhao; Huayang Wuhan University Huang; Miao Wuhan University Li; Yu Wuhan University Wu
  Language-conditioned robotic learning has significantly enhanced robot
adaptability by enabling a single model to execute diverse tasks in response to
verbal commands. Despite these advancements, security vulnerabilities within
this domain remain largely unexplored. This paper addresses this gap by
proposing a novel adversarial prompt attack tailored to language-conditioned
robotic models. Our approach involves crafting a universal adversarial prefix
that induces the model to perform unintended actions when added to any original
prompt. We demonstrate that existing adversarial techniques exhibit limited
effectiveness when directly transferred to the robotic domain due to the
inherent robustness of discretized robotic action spaces. To overcome this
challenge, we propose to optimize adversarial prefixes based on continuous
action representations, circumventing the discretization process. Additionally,
we identify the beneficial impact of intermediate features on adversarial
attacks and leverage the negative gradient of intermediate self-attention
features to further enhance attack efficacy. Extensive experiments on VIMA
models across 13 robot manipulation tasks validate the superiority of our
method over existing approaches and demonstrate its transferability across
different model variants.


http://arxiv.org/abs/2411.13553
AI-generated Image Detection: Passive or Watermark? (31%)
Moyang Guo; Yuepeng Hu; Zhengyuan Jiang; Zeyu Li; Amir Sadovnik; Arka Daw; Neil Gong
  While text-to-image models offer numerous benefits, they also pose
significant societal risks. Detecting AI-generated images is crucial for
mitigating these risks. Detection methods can be broadly categorized into
passive and watermark-based approaches: passive detectors rely on artifacts
present in AI-generated images, whereas watermark-based detectors proactively
embed watermarks into such images. A key question is which type of detector
performs better in terms of effectiveness, robustness, and efficiency. However,
the current literature lacks a comprehensive understanding of this issue. In
this work, we aim to bridge that gap by developing ImageDetectBench, the first
comprehensive benchmark to compare the effectiveness, robustness, and
efficiency of passive and watermark-based detectors. Our benchmark includes
four datasets, each containing a mix of AI-generated and non-AI-generated
images. We evaluate five passive detectors and four watermark-based detectors
against eight types of common perturbations and three types of adversarial
perturbations. Our benchmark results reveal several interesting findings. For
instance, watermark-based detectors consistently outperform passive detectors,
both in the presence and absence of perturbations. Based on these insights, we
provide recommendations for detecting AI-generated images, e.g., when both
types of detectors are applicable, watermark-based detectors should be the
preferred choice. Our code and data are publicly available at
https://github.com/moyangkuo/ImageDetectBench.git.


http://arxiv.org/abs/2411.13459
SoK: A Systems Perspective on Compound AI Threats and Countermeasures. (12%)
Sarbartha Banerjee; Prateek Sahu; Mulong Luo; Anjo Vahldiek-Oberwagner; Neeraja J. Yadwadkar; Mohit Tiwari
  Large language models (LLMs) used across enterprises often use proprietary
models and operate on sensitive inputs and data. The wide range of attack
vectors identified in prior research - targeting various software and hardware
components used in training and inference - makes it extremely challenging to
enforce confidentiality and integrity policies.
  As we advance towards constructing compound AI inference pipelines that
integrate multiple large language models (LLMs), the attack surfaces expand
significantly. Attackers now focus on the AI algorithms as well as the software
and hardware components associated with these systems. While current research
often examines these elements in isolation, we find that combining cross-layer
attack observations can enable powerful end-to-end attacks with minimal
assumptions about the threat model. Given, the sheer number of existing attacks
at each layer, we need a holistic and systemized understanding of different
attack vectors at each layer.
  This SoK discusses different software and hardware attacks applicable to
compound AI systems and demonstrates how combining multiple attack mechanisms
can reduce the threat model assumptions required for an isolated attack. Next,
we systematize the ML attacks in lines with the Mitre Att&ck framework to
better position each attack based on the threat model. Finally, we outline the
existing countermeasures for both software and hardware layers and discuss the
necessity of a comprehensive defense strategy to enable the secure and
high-performance deployment of compound AI systems.


http://arxiv.org/abs/2411.13144
CopyrightMeter: Revisiting Copyright Protection in Text-to-image Models. (12%)
Naen Xu; Changjiang Li; Tianyu Du; Minxi Li; Wenjie Luo; Jiacheng Liang; Yuyuan Li; Xuhong Zhang; Meng Han; Jianwei Yin; Ting Wang
  Text-to-image diffusion models have emerged as powerful tools for generating
high-quality images from textual descriptions. However, their increasing
popularity has raised significant copyright concerns, as these models can be
misused to reproduce copyrighted content without authorization. In response,
recent studies have proposed various copyright protection methods, including
adversarial perturbation, concept erasure, and watermarking techniques.
However, their effectiveness and robustness against advanced attacks remain
largely unexplored. Moreover, the lack of unified evaluation frameworks has
hindered systematic comparison and fair assessment of different approaches. To
bridge this gap, we systematize existing copyright protection methods and
attacks, providing a unified taxonomy of their design spaces. We then develop
CopyrightMeter, a unified evaluation framework that incorporates 17
state-of-the-art protections and 16 representative attacks. Leveraging
CopyrightMeter, we comprehensively evaluate protection methods across multiple
dimensions, thereby uncovering how different design choices impact fidelity,
efficacy, and resilience under attacks. Our analysis reveals several key
findings: (i) most protections (16/17) are not resilient against attacks; (ii)
the "best" protection varies depending on the target priority; (iii) more
advanced attacks significantly promote the upgrading of protections. These
insights provide concrete guidance for developing more robust protection
methods, while its unified evaluation protocol establishes a standard benchmark
for future copyright protection research in text-to-image generation.


http://arxiv.org/abs/2411.13047
Bounding-box Watermarking: Defense against Model Extraction Attacks on Object Detectors. (5%)
Satoru Koda; Ikuya Morikawa
  Deep neural networks (DNNs) deployed in a cloud often allow users to query
models via the APIs. However, these APIs expose the models to model extraction
attacks (MEAs). In this attack, the attacker attempts to duplicate the target
model by abusing the responses from the API. Backdoor-based DNN watermarking is
known as a promising defense against MEAs, wherein the defender injects a
backdoor into extracted models via API responses. The backdoor is used as a
watermark of the model; if a suspicious model has the watermark (i.e.,
backdoor), it is verified as an extracted model. This work focuses on object
detection (OD) models. Existing backdoor attacks on OD models are not
applicable for model watermarking as the defense against MEAs on a realistic
threat model. Our proposed approach involves inserting a backdoor into
extracted models via APIs by stealthily modifying the bounding-boxes (BBs) of
objects detected in queries while keeping the OD capability. In our experiments
on three OD datasets, the proposed approach succeeded in identifying the
extracted models with 100% accuracy in a wide variety of experimental
scenarios.


http://arxiv.org/abs/2411.13425
WaterPark: A Robustness Assessment of Language Model Watermarking. (1%)
Jiacheng Liang; Zian Wang; Lauren Hong; Shouling Ji; Ting Wang
  Various watermarking methods (``watermarkers'') have been proposed to
identify LLM-generated texts; yet, due to the lack of unified evaluation
platforms, many critical questions remain under-explored: i) What are the
strengths/limitations of various watermarkers, especially their attack
robustness? ii) How do various design choices impact their robustness? iii) How
to optimally operate watermarkers in adversarial environments? To fill this
gap, we systematize existing LLM watermarkers and watermark removal attacks,
mapping out their design spaces. We then develop WaterPark, a unified platform
that integrates 10 state-of-the-art watermarkers and 12 representative attacks.
More importantly, by leveraging WaterPark, we conduct a comprehensive
assessment of existing watermarkers, unveiling the impact of various design
choices on their attack robustness. We further explore the best practices to
operate watermarkers in adversarial environments. We believe our study sheds
light on current LLM watermarking techniques while WaterPark serves as a
valuable testbed to facilitate future research.


http://arxiv.org/abs/2411.12473
NMT-Obfuscator Attack: Ignore a sentence in translation with only one word. (99%)
Sahar Sadrizadeh; César Descalzo; Ljiljana Dolamic; Pascal Frossard
  Neural Machine Translation systems are used in diverse applications due to
their impressive performance. However, recent studies have shown that these
systems are vulnerable to carefully crafted small perturbations to their
inputs, known as adversarial attacks. In this paper, we propose a new type of
adversarial attack against NMT models. In this attack, we find a word to be
added between two sentences such that the second sentence is ignored and not
translated by the NMT model. The word added between the two sentences is such
that the whole adversarial text is natural in the source language. This type of
attack can be harmful in practical scenarios since the attacker can hide
malicious information in the automatic translation made by the target NMT
model. Our experiments show that different NMT models and translation tasks are
vulnerable to this type of attack. Our attack can successfully force the NMT
models to ignore the second part of the input in the translation for more than
50% of all cases while being able to maintain low perplexity for the whole
input.


http://arxiv.org/abs/2411.12575
Stochastic BIQA: Median Randomized Smoothing for Certified Blind Image Quality Assessment. (75%)
Ekaterina Shumitskaya; Mikhail Pautov; Dmitriy Vatolin; Anastasia Antsiferova
  Most modern No-Reference Image-Quality Assessment (NR-IQA) metrics are based
on neural networks vulnerable to adversarial attacks. Attacks on such metrics
lead to incorrect image/video quality predictions, which poses significant
risks, especially in public benchmarks. Developers of image processing
algorithms may unfairly increase the score of a target IQA metric without
improving the actual quality of the adversarial image. Although some empirical
defenses for IQA metrics were proposed, they do not provide theoretical
guarantees and may be vulnerable to adaptive attacks. This work focuses on
developing a provably robust no-reference IQA metric. Our method is based on
Median Smoothing (MS) combined with an additional convolution denoiser with
ranking loss to improve the SROCC and PLCC scores of the defended IQA metric.
Compared with two prior methods on three datasets, our method exhibited
superior SROCC and PLCC scores while maintaining comparable certified
guarantees.


http://arxiv.org/abs/2411.12701
When Backdoors Speak: Understanding LLM Backdoor Attacks Through Model-Generated Explanations. (3%)
Huaizhi Ge; Yiming Li; Qifan Wang; Yongfeng Zhang; Ruixiang Tang
  Large Language Models (LLMs) are vulnerable to backdoor attacks, where hidden
triggers can maliciously manipulate model behavior. While several backdoor
attack methods have been proposed, the mechanisms by which backdoor functions
operate in LLMs remain underexplored. In this paper, we move beyond attacking
LLMs and investigate backdoor functionality through the novel lens of natural
language explanations. Specifically, we leverage LLMs' generative capabilities
to produce human-understandable explanations for their decisions, allowing us
to compare explanations for clean and poisoned samples. We explore various
backdoor attacks and embed the backdoor into LLaMA models for multiple tasks.
Our experiments show that backdoored models produce higher-quality explanations
for clean data compared to poisoned data, while generating significantly more
consistent explanations for poisoned data than for clean data. We further
analyze the explanation generation process, revealing that at the token level,
the explanation token of poisoned samples only appears in the final few
transformer layers of the LLM. At the sentence level, attention dynamics
indicate that poisoned inputs shift attention from the input context when
generating the explanation. These findings deepen our understanding of backdoor
attack mechanisms in LLMs and offer a framework for detecting such
vulnerabilities through explainability techniques, contributing to the
development of more secure LLMs.


http://arxiv.org/abs/2411.12071
Theoretical Corrections and the Leveraging of Reinforcement Learning to Enhance Triangle Attack. (99%)
Nicole Meng; Caleb Manicke; David Chen; Yingjie Lao; Caiwen Ding; Pengyu Hong; Kaleel Mahmood
  Adversarial examples represent a serious issue for the application of machine
learning models in many sensitive domains. For generating adversarial examples,
decision based black-box attacks are one of the most practical techniques as
they only require query access to the model. One of the most recently proposed
state-of-the-art decision based black-box attacks is Triangle Attack (TA). In
this paper, we offer a high-level description of TA and explain potential
theoretical limitations. We then propose a new decision based black-box attack,
Triangle Attack with Reinforcement Learning (TARL). Our new attack addresses
the limits of TA by leveraging reinforcement learning. This creates an attack
that can achieve similar, if not better, attack accuracy than TA with half as
many queries on state-of-the-art classifiers and defenses across ImageNet and
CIFAR-10.


http://arxiv.org/abs/2411.11389
Adapting to Cyber Threats: A Phishing Evolution Network (PEN) Framework for Phishing Generation and Analyzing Evolution Patterns using Large Language Models. (87%)
Fengchao Chen; Tingmin Wu; Van Nguyen; Shuo Wang; Hongsheng Hu; Alsharif Abuadbba; Carsten Rudolph
  Phishing remains a pervasive cyber threat, as attackers craft deceptive
emails to lure victims into revealing sensitive information. While Artificial
Intelligence (AI), particularly deep learning, has become a key component in
defending against phishing attacks, these approaches face critical limitations.
The scarcity of publicly available, diverse, and updated data, largely due to
privacy concerns, constrains their effectiveness. As phishing tactics evolve
rapidly, models trained on limited, outdated data struggle to detect new,
sophisticated deception strategies, leaving systems vulnerable to an
ever-growing array of attacks. Addressing this gap is essential to
strengthening defenses in an increasingly hostile cyber landscape. To address
this gap, we propose the Phishing Evolution Network (PEN), a framework
leveraging large language models (LLMs) and adversarial training mechanisms to
continuously generate high quality and realistic diverse phishing samples, and
analyze features of LLM-provided phishing to understand evolving phishing
patterns. We evaluate the quality and diversity of phishing samples generated
by PEN and find that it produces over 80% realistic phishing samples,
effectively expanding phishing datasets across seven dominant types. These
PEN-generated samples enhance the performance of current phishing detectors,
leading to a 40% improvement in detection accuracy. Additionally, the use of
PEN significantly boosts model robustness, reducing detectors' sensitivity to
perturbations by up to 60%, thereby decreasing attack success rates under
adversarial conditions. When we analyze the phishing patterns that are used in
LLM-generated phishing, the cognitive complexity and the tone of time
limitation are detected with statistically significant differences compared
with existing phishing.


http://arxiv.org/abs/2411.12220
DeTrigger: A Gradient-Centric Approach to Backdoor Attack Mitigation in Federated Learning. (75%)
Kichang Lee; Yujin Shin; Jonghyuk Yun; Jun Han; JeongGil Ko
  Federated Learning (FL) enables collaborative model training across
distributed devices while preserving local data privacy, making it ideal for
mobile and embedded systems. However, the decentralized nature of FL also opens
vulnerabilities to model poisoning attacks, particularly backdoor attacks,
where adversaries implant trigger patterns to manipulate model predictions. In
this paper, we propose DeTrigger, a scalable and efficient backdoor-robust
federated learning framework that leverages insights from adversarial attack
methodologies. By employing gradient analysis with temperature scaling,
DeTrigger detects and isolates backdoor triggers, allowing for precise model
weight pruning of backdoor activations without sacrificing benign model
knowledge. Extensive evaluations across four widely used datasets demonstrate
that DeTrigger achieves up to 251x faster detection than traditional methods
and mitigates backdoor attacks by up to 98.9%, with minimal impact on global
model accuracy. Our findings establish DeTrigger as a robust and scalable
solution to protect federated learning environments against sophisticated
backdoor threats.


http://arxiv.org/abs/2411.12768
CROW: Eliminating Backdoors from Large Language Models via Internal Consistency Regularization. (67%)
Nay Myat Min; Long H. Pham; Yige Li; Jun Sun
  Recent studies reveal that Large Language Models (LLMs) are susceptible to
backdoor attacks, where adversaries embed hidden triggers that manipulate model
responses. Existing backdoor defense methods are primarily designed for vision
or classification tasks, and are thus ineffective for text generation tasks,
leaving LLMs vulnerable. We introduce Internal Consistency Regularization
(CROW), a novel defense using consistency regularization finetuning to address
layer-wise inconsistencies caused by backdoor triggers. CROW leverages the
intuition that clean models exhibit smooth, consistent transitions in hidden
representations across layers, whereas backdoored models show noticeable
fluctuation when triggered. By enforcing internal consistency through
adversarial perturbations and regularization, CROW neutralizes backdoor effects
without requiring clean reference models or prior trigger knowledge, relying
only on a small set of clean data. This makes it practical for deployment
across various LLM architectures. Experimental results demonstrate that CROW
consistently achieves a significant reductions in attack success rates across
diverse backdoor strategies and tasks, including negative sentiment, targeted
refusal, and code injection, on models such as Llama-2 (7B, 13B), CodeLlama
(7B, 13B) and Mistral-7B, while preserving the model's generative capabilities.


http://arxiv.org/abs/2411.11525
Reliable Poisoned Sample Detection against Backdoor Attacks Enhanced by Sharpness Aware Minimization. (50%)
Mingda Zhang; Mingli Zhu; Zihao Zhu; Baoyuan Wu
  Backdoor attack has been considered as a serious security threat to deep
neural networks (DNNs). Poisoned sample detection (PSD) that aims at filtering
out poisoned samples from an untrustworthy training dataset has shown very
promising performance for defending against data poisoning based backdoor
attacks. However, we observe that the detection performance of many advanced
methods is likely to be unstable when facing weak backdoor attacks, such as low
poisoning ratio or weak trigger strength. To further verify this observation,
we make a statistical investigation among various backdoor attacks and poisoned
sample detections, showing a positive correlation between backdoor effect and
detection performance. It inspires us to strengthen the backdoor effect to
enhance detection performance. Since we cannot achieve that goal via directly
manipulating poisoning ratio or trigger strength, we propose to train one model
using the Sharpness-Aware Minimization (SAM) algorithm, rather than the vanilla
training algorithm. We also provide both empirical and theoretical analysis
about how SAM training strengthens the backdoor effect. Then, this SAM trained
model can be seamlessly integrated with any off-the-shelf PSD method that
extracts discriminative features from the trained model for detection, called
SAM-enhanced PSD. Extensive experiments on several benchmark datasets show the
reliable detection performance of the proposed method against both weak and
strong backdoor attacks, with significant improvements against various attacks
($+34.38\%$ TPR on average), over the conventional PSD methods (i.e., without
SAM enhancement). Overall, this work provides new insights about PSD and
proposes a novel approach that can complement existing detection methods, which
may inspire more in-depth explorations in this field.


http://arxiv.org/abs/2411.11677
Few-shot Model Extraction Attacks against Sequential Recommender Systems. (38%)
Hui Zhang; Fu Liu
  Among adversarial attacks against sequential recommender systems, model
extraction attacks represent a method to attack sequential recommendation
models without prior knowledge. Existing research has primarily concentrated on
the adversary's execution of black-box attacks through data-free model
extraction. However, a significant gap remains in the literature concerning the
development of surrogate models by adversaries with access to few-shot raw data
(10\% even less). That is, the challenge of how to construct a surrogate model
with high functional similarity within the context of few-shot data scenarios
remains an issue that requires resolution.This study addresses this gap by
introducing a novel few-shot model extraction framework against sequential
recommenders, which is designed to construct a superior surrogate model with
the utilization of few-shot data. The proposed few-shot model extraction
framework is comprised of two components: an autoregressive augmentation
generation strategy and a bidirectional repair loss-facilitated model
distillation procedure. Specifically, to generate synthetic data that closely
approximate the distribution of raw data, autoregressive augmentation
generation strategy integrates a probabilistic interaction sampler to extract
inherent dependencies and a synthesis determinant signal module to characterize
user behavioral patterns. Subsequently, bidirectional repair loss, which target
the discrepancies between the recommendation lists, is designed as auxiliary
loss to rectify erroneous predictions from surrogate models, transferring
knowledge from the victim model to the surrogate model effectively. Experiments
on three datasets show that the proposed few-shot model extraction framework
yields superior surrogate models.


http://arxiv.org/abs/2411.11434
CLUE-MARK: Watermarking Diffusion Models using CLWE. (26%)
Kareem Shehata; Aashish Kolluri; Prateek Saxena
  As AI-generated images become widespread, reliable watermarking is essential
for content verification, copyright enforcement, and combating disinformation.
Existing techniques rely on heuristic approaches and lack formal guarantees of
undetectability, making them vulnerable to steganographic attacks that can
expose or erase the watermark. Additionally, these techniques often degrade
output quality by introducing perceptible changes, which is not only
undesirable but an important barrier to adoption in practice.
  In this work, we introduce CLUE-Mark, the first provably undetectable
watermarking scheme for diffusion models. CLUE-Mark requires no changes to the
model being watermarked, is computationally efficient, and because it is
provably undetectable is guaranteed to have no impact on model output quality.
Our approach leverages the Continuous Learning With Errors (CLWE) problem -- a
cryptographically hard lattice problem -- to embed watermarks in the latent
noise vectors used by diffusion models. By proving undetectability via
reduction to a cryptographically hard problem we ensure not only that the
watermark is imperceptible to human observers or adhoc heuristics, but to
\emph{any} efficient detector that does not have the secret key. CLUE-Mark
allows multiple keys to be embedded, enabling traceability of images to
specific users without altering model parameters. Empirical evaluations on
state-of-the-art diffusion models confirm that CLUE-Mark achieves high
recoverability, preserves image quality, and is robust to minor perturbations
such JPEG compression and brightness adjustments. Uniquely, CLUE-Mark cannot be
detected nor removed by recent steganographic attacks.


http://arxiv.org/abs/2411.11407
The Dark Side of Trust: Authority Citation-Driven Jailbreak Attacks on Large Language Models. (13%)
Xikang Yang; Xuehai Tang; Jizhong Han; Songlin Hu
  The widespread deployment of large language models (LLMs) across various
domains has showcased their immense potential while exposing significant safety
vulnerabilities. A major concern is ensuring that LLM-generated content aligns
with human values. Existing jailbreak techniques reveal how this alignment can
be compromised through specific prompts or adversarial suffixes. In this study,
we introduce a new threat: LLMs' bias toward authority. While this inherent
bias can improve the quality of outputs generated by LLMs, it also introduces a
potential vulnerability, increasing the risk of producing harmful content.
Notably, the biases in LLMs is the varying levels of trust given to different
types of authoritative information in harmful queries. For example, malware
development often favors trust GitHub. To better reveal the risks with LLM, we
propose DarkCite, an adaptive authority citation matcher and generator designed
for a black-box setting. DarkCite matches optimal citation types to specific
risk types and generates authoritative citations relevant to harmful
instructions, enabling more effective jailbreak attacks on aligned LLMs.Our
experiments show that DarkCite achieves a higher attack success rate (e.g.,
LLama-2 at 76% versus 68%) than previous methods. To counter this risk, we
propose an authenticity and harm verification defense strategy, raising the
average defense pass rate (DPR) from 11% to 74%. More importantly, the ability
to link citations to the content they encompass has become a foundational
function in LLMs, amplifying the influence of LLMs' bias toward authority.


http://arxiv.org/abs/2411.11795
Exploring adversarial robustness of JPEG AI: methodology, comparison and new methods. (8%)
Egor Kovalev; Georgii Bychkov; Khaled Abud; Aleksandr Gushchin; Anna Chistyakova; Sergey Lavrushkin; Dmitriy Vatolin; Anastasia Antsiferova
  Adversarial robustness of neural networks is an increasingly important area
of research, combining studies on computer vision models, large language models
(LLMs), and others. With the release of JPEG AI - the first standard for
end-to-end neural image compression (NIC) methods - the question of its
robustness has become critically significant. JPEG AI is among the first
international, real-world applications of neural-network-based models to be
embedded in consumer devices. However, research on NIC robustness has been
limited to open-source codecs and a narrow range of attacks. This paper
proposes a new methodology for measuring NIC robustness to adversarial attacks.
We present the first large-scale evaluation of JPEG AI's robustness, comparing
it with other NIC models. Our evaluation results and code are publicly
available online (link is hidden for a blind review).


http://arxiv.org/abs/2411.13587
Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Robotics. (86%)
Taowen Wang; Dongfang Liu; James Chenhao Liang; Wenhao Yang; Qifan Wang; Cheng Han; Jiebo Luo; Ruixiang Tang
  Recently in robotics, Vision-Language-Action (VLA) models have emerged as a
transformative approach, enabling robots to execute complex tasks by
integrating visual and linguistic inputs within an end-to-end learning
framework. While VLA models offer significant capabilities, they also introduce
new attack surfaces, making them vulnerable to adversarial attacks. With these
vulnerabilities largely unexplored, this paper systematically quantifies the
robustness of VLA-based robotic systems. Recognizing the unique demands of
robotic execution, our attack objectives target the inherent spatial and
functional characteristics of robotic systems. In particular, we introduce an
untargeted position-aware attack objective that leverages spatial foundations
to destabilize robotic actions, and a targeted attack objective that
manipulates the robotic trajectory. Additionally, we design an adversarial
patch generation approach that places a small, colorful patch within the
camera's view, effectively executing the attack in both digital and physical
environments. Our evaluation reveals a marked degradation in task success
rates, with up to a 100\% reduction across a suite of simulated robotic tasks,
highlighting critical security gaps in current VLA architectures. By unveiling
these vulnerabilities and proposing actionable evaluation metrics, this work
advances both the understanding and enhancement of safety for VLA-based robotic
systems, underscoring the necessity for developing robust defense strategies
prior to physical-world deployments.


http://arxiv.org/abs/2411.11114
JailbreakLens: Interpreting Jailbreak Mechanism in the Lens of Representation and Circuit. (47%)
Zeqing He; Zhibo Wang; Zhixuan Chu; Huiyu Xu; Rui Zheng; Kui Ren; Chun Chen
  Despite the outstanding performance of Large language models (LLMs) in
diverse tasks, they are vulnerable to jailbreak attacks, wherein adversarial
prompts are crafted to bypass their security mechanisms and elicit unexpected
responses.Although jailbreak attacks are prevalent, the understanding of their
underlying mechanisms remains limited. Recent studies have explain typical
jailbreaking behavior (e.g., the degree to which the model refuses to respond)
of LLMs by analyzing the representation shifts in their latent space caused by
jailbreak prompts or identifying key neurons that contribute to the success of
these attacks. However, these studies neither explore diverse jailbreak
patterns nor provide a fine-grained explanation from the failure of circuit to
the changes of representational, leaving significant gaps in uncovering the
jailbreak mechanism. In this paper, we propose JailbreakLens, an interpretation
framework that analyzes jailbreak mechanisms from both representation (which
reveals how jailbreaks alter the model's harmfulness perception) and circuit
perspectives (which uncovers the causes of these deceptions by identifying key
circuits contributing to the vulnerability), tracking their evolution
throughout the entire response generation process. We then conduct an in-depth
evaluation of jailbreak behavior on four mainstream LLMs under seven jailbreak
strategies. Our evaluation finds that jailbreak prompts amplify components that
reinforce affirmative responses while suppressing those that produce refusal.
Although this manipulation shifts model representations toward safe clusters to
deceive the LLM, leading it to provide detailed responses instead of refusals,
it still produce abnormal activation which can be caught in the circuit
analysis.


http://arxiv.org/abs/2411.11200
Countering Backdoor Attacks in Image Recognition: A Survey and Evaluation of Mitigation Strategies. (22%)
Kealan Dunnett; Reza Arablouei; Dimity Miller; Volkan Dedeoglu; Raja Jurdak
  The widespread adoption of deep learning across various industries has
introduced substantial challenges, particularly in terms of model
explainability and security. The inherent complexity of deep learning models,
while contributing to their effectiveness, also renders them susceptible to
adversarial attacks. Among these, backdoor attacks are especially concerning,
as they involve surreptitiously embedding specific triggers within training
data, causing the model to exhibit aberrant behavior when presented with input
containing the triggers. Such attacks often exploit vulnerabilities in
outsourced processes, compromising model integrity without affecting
performance on clean (trigger-free) input data. In this paper, we present a
comprehensive review of existing mitigation strategies designed to counter
backdoor attacks in image recognition. We provide an in-depth analysis of the
theoretical foundations, practical efficacy, and limitations of these
approaches. In addition, we conduct an extensive benchmarking of sixteen
state-of-the-art approaches against eight distinct backdoor attacks, utilizing
three datasets, four model architectures, and three poisoning ratios. Our
results, derived from 122,236 individual experiments, indicate that while many
approaches provide some level of protection, their performance can vary
considerably. Furthermore, when compared to two seminal approaches, most newer
approaches do not demonstrate substantial improvements in overall performance
or consistency across diverse settings. Drawing from these findings, we propose
potential directions for developing more effective and generalizable defensive
mechanisms in the future.


http://arxiv.org/abs/2411.11195
SoK: Unifying Cybersecurity and Cybersafety of Multimodal Foundation Models with an Information Theory Approach. (9%)
Ruoxi Sun; Jiamin Chang; Hammond Pearce; Chaowei Xiao; Bo Li; Qi Wu; Surya Nepal; Minhui Xue
  Multimodal foundation models (MFMs) represent a significant advancement in
artificial intelligence, combining diverse data modalities to enhance learning
and understanding across a wide range of applications. However, this
integration also brings unique safety and security challenges. In this paper,
we conceptualize cybersafety and cybersecurity in the context of multimodal
learning and present a comprehensive Systematization of Knowledge (SoK) to
unify these concepts in MFMs, identifying key threats to these models. We
propose a taxonomy framework grounded in information theory, evaluating and
categorizing threats through the concepts of channel capacity, signal, noise,
and bandwidth. This approach provides a novel framework that unifies model
safety and system security in MFMs, offering a more comprehensive and
actionable understanding of the risks involved. We used this to explore
existing defense mechanisms, and identified gaps in current research -
particularly, a lack of protection for alignment between modalities and a need
for more systematic defense methods. Our work contributes to a deeper
understanding of the security and safety landscape in MFMs, providing
researchers and practitioners with valuable insights for improving the
robustness and reliability of these models.


http://arxiv.org/abs/2411.11144
CLMIA: Membership Inference Attacks via Unsupervised Contrastive Learning. (2%)
Depeng School of Computer Science and Technology, Anhui University Chen; Xiao School of Computer Science and Technology, Anhui University Liu; Jie School of Computer Science and Technology, Anhui University Cui; Hong School of Computer Science and Technology, Anhui University Zhong
  Since machine learning model is often trained on a limited data set, the
model is trained multiple times on the same data sample, which causes the model
to memorize most of the training set data. Membership Inference Attacks (MIAs)
exploit this feature to determine whether a data sample is used for training a
machine learning model. However, in realistic scenarios, it is difficult for
the adversary to obtain enough qualified samples that mark accurate identity
information, especially since most samples are non-members in real world
applications. To address this limitation, in this paper, we propose a new
attack method called CLMIA, which uses unsupervised contrastive learning to
train an attack model without using extra membership status information.
Meanwhile, in CLMIA, we require only a small amount of data with known
membership status to fine-tune the attack model. Experimental results
demonstrate that CLMIA performs better than existing attack methods for
different datasets and model structures, especially with data with less marked
identity information. In addition, we experimentally find that the attack
performs differently for different proportions of labeled identity information
for member and non-member data. More analysis proves that our attack method
performs better with less labeled identity information, which applies to more
realistic scenarios.


http://arxiv.org/abs/2411.10174
A Hard-Label Cryptanalytic Extraction of Non-Fully Connected Deep Neural Networks using Side-Channel Attacks. (98%)
Benoit Coqueret; Mathieu Carbone; Olivier Sentieys; Gabriel Zaid
  During the past decade, Deep Neural Networks (DNNs) proved their value on a
large variety of subjects. However despite their high value and public
accessibility, the protection of the intellectual property of DNNs is still an
issue and an emerging research field. Recent works have successfully extracted
fully-connected DNNs using cryptanalytic methods in hard-label settings,
proving that it was possible to copy a DNN with high fidelity, i.e., high
similitude in the output predictions. However, the current cryptanalytic
attacks cannot target complex, i.e., not fully connected, DNNs and are limited
to special cases of neurons present in deep networks.
  In this work, we introduce a new end-to-end attack framework designed for
model extraction of embedded DNNs with high fidelity. We describe a new
black-box side-channel attack which splits the DNN in several linear parts for
which we can perform cryptanalytic extraction and retrieve the weights in
hard-label settings. With this method, we are able to adapt cryptanalytic
extraction, for the first time, to non-fully connected DNNs, while maintaining
a high fidelity. We validate our contributions by targeting several
architectures implemented on a microcontroller unit, including a Multi-Layer
Perceptron (MLP) of 1.7 million parameters and a shortened MobileNetv1. Our
framework successfully extracts all of these DNNs with high fidelity (88.4% for
the MobileNetv1 and 93.2% for the MLP). Furthermore, we use the stolen model to
generate adversarial examples and achieve close to white-box performance on the
victim's model (95.8% and 96.7% transfer rate).


http://arxiv.org/abs/2411.10500
Edge-Only Universal Adversarial Attacks in Distributed Learning. (98%)
Giulio Rossolini; Tommaso Baldi; Alessandro Biondi; Giorgio Buttazzo
  Distributed learning frameworks, which partition neural network models across
multiple computing nodes, enhance efficiency in collaborative edge-cloud
systems but may also introduce new vulnerabilities. In this work, we explore
the feasibility of generating universal adversarial attacks when an attacker
has access to the edge part of the model only, which consists in the first
network layers. Unlike traditional universal adversarial perturbations (UAPs)
that require full model knowledge, our approach shows that adversaries can
induce effective mispredictions in the unknown cloud part by leveraging key
features on the edge side. Specifically, we train lightweight classifiers from
intermediate features available at the edge, i.e., before the split point, and
use them in a novel targeted optimization to craft effective UAPs. Our results
on ImageNet demonstrate strong attack transferability to the unknown cloud
part. Additionally, we analyze the capability of an attacker to achieve
targeted adversarial effect with edge-only knowledge, revealing intriguing
behaviors. By introducing the first adversarial attacks with edge-only
knowledge in split inference, this work underscores the importance of
addressing partial model access in adversarial robustness, encouraging further
research in this area.


http://arxiv.org/abs/2411.10498
Prompt-Guided Environmentally Consistent Adversarial Patch. (82%)
Chaoqun Li; Huanqian Yan; Lifeng Zhou; Tairan Chen; Zhuodong Liu; Hang Su
  Adversarial attacks in the physical world pose a significant threat to the
security of vision-based systems, such as facial recognition and autonomous
driving. Existing adversarial patch methods primarily focus on improving attack
performance, but they often produce patches that are easily detectable by
humans and struggle to achieve environmental consistency, i.e., blending
patches into the environment. This paper introduces a novel approach for
generating adversarial patches, which addresses both the visual naturalness and
environmental consistency of the patches. We propose Prompt-Guided
Environmentally Consistent Adversarial Patch (PG-ECAP), a method that aligns
the patch with the environment to ensure seamless integration into the
environment. The approach leverages diffusion models to generate patches that
are both environmental consistency and effective in evading detection. To
further enhance the naturalness and consistency, we introduce two alignment
losses: Prompt Alignment Loss and Latent Space Alignment Loss, ensuring that
the generated patch maintains its adversarial properties while fitting
naturally within its environment. Extensive experiments in both digital and
physical domains demonstrate that PG-ECAP outperforms existing methods in
attack success rate and environmental consistency.


http://arxiv.org/abs/2411.10367
Continual Adversarial Reinforcement Learning (CARL) of False Data Injection detection: forgetting and explainability. (81%)
Pooja Aslami; Kejun Chen; Timothy M. Hansen; Malik Hassanaly
  False data injection attacks (FDIAs) on smart inverters are a growing concern
linked to increased renewable energy production. While data-based FDIA
detection methods are also actively developed, we show that they remain
vulnerable to impactful and stealthy adversarial examples that can be crafted
using Reinforcement Learning (RL). We propose to include such adversarial
examples in data-based detection training procedure via a continual adversarial
RL (CARL) approach. This way, one can pinpoint the deficiencies of data-based
detection, thereby offering explainability during their incremental
improvement. We show that a continual learning implementation is subject to
catastrophic forgetting, and additionally show that forgetting can be addressed
by employing a joint training strategy on all generated FDIA scenarios.


http://arxiv.org/abs/2411.10034
EveGuard: Defeating Vibration-based Side-Channel Eavesdropping with Audio Adversarial Perturbations. (68%)
Jung-Woo Chang; Ke Sun; David Xia; Xinyu Zhang; Farinaz Koushanfar
  Vibrometry-based side channels pose a significant privacy risk, exploiting
sensors like mmWave radars, light sensors, and accelerometers to detect
vibrations from sound sources or proximate objects, enabling speech
eavesdropping. Despite various proposed defenses, these involve costly hardware
solutions with inherent physical limitations. This paper presents EveGuard, a
software-driven defense framework that creates adversarial audio, protecting
voice privacy from side channels without compromising human perception. We
leverage the distinct sensing capabilities of side channels and traditional
microphones where side channels capture vibrations and microphones record
changes in air pressure, resulting in different frequency responses. EveGuard
first proposes a perturbation generator model (PGM) that effectively suppresses
sensor-based eavesdropping while maintaining high audio quality. Second, to
enable end-to-end training of PGM, we introduce a new domain translation task
called Eve-GAN for inferring an eavesdropped signal from a given audio. We
further apply few-shot learning to mitigate the data collection overhead for
Eve-GAN training. Our extensive experiments show that EveGuard achieves a
protection rate of more than 97 percent from audio classifiers and
significantly hinders eavesdropped audio reconstruction. We further validate
the performance of EveGuard across three adaptive attack mechanisms. We have
conducted a user study to verify the perceptual quality of our perturbed audio.


http://arxiv.org/abs/2411.10565
Comparing Robustness Against Adversarial Attacks in Code Generation: LLM-Generated vs. Human-Written. (68%)
Md Abdul Awal; Mrigank Rochan; Chanchal K. Roy
  Thanks to the widespread adoption of Large Language Models (LLMs) in software
engineering research, the long-standing dream of automated code generation has
become a reality on a large scale. Nowadays, LLMs such as GitHub Copilot and
ChatGPT are extensively used in code generation for enterprise and open-source
software development and maintenance. Despite their unprecedented successes in
code generation, research indicates that codes generated by LLMs exhibit
vulnerabilities and security issues. Several studies have been conducted to
evaluate code generated by LLMs, considering various aspects such as security,
vulnerability, code smells, and robustness. While some studies have compared
the performance of LLMs with that of humans in various software engineering
tasks, there's a notable gap in research: no studies have directly compared
human-written and LLM-generated code for their robustness analysis. To fill
this void, this paper introduces an empirical study to evaluate the adversarial
robustness of Pre-trained Models of Code (PTMCs) fine-tuned on code written by
humans and generated by LLMs against adversarial attacks for software clone
detection. These attacks could potentially undermine software security and
reliability. We consider two datasets, two state-of-the-art PTMCs, two
robustness evaluation criteria, and three metrics to use in our experiments.
Regarding effectiveness criteria, PTMCs fine-tuned on human-written code always
demonstrate more robustness than those fine-tuned on LLMs-generated code. On
the other hand, in terms of adversarial code quality, in 75% experimental
combinations, PTMCs fine-tuned on the human-written code exhibit more
robustness than the PTMCs fine-tuned on the LLMs-generated code.


http://arxiv.org/abs/2411.10329
Safe Text-to-Image Generation: Simply Sanitize the Prompt Embedding. (11%)
Huming Qiu; Guanxu Chen; Mi Zhang; Min Yang
  In recent years, text-to-image (T2I) generation models have made significant
progress in generating high-quality images that align with text descriptions.
However, these models also face the risk of unsafe generation, potentially
producing harmful content that violates usage policies, such as explicit
material. Existing safe generation methods typically focus on suppressing
inappropriate content by erasing undesired concepts from visual
representations, while neglecting to sanitize the textual representation.
Although these methods help mitigate the risk of misuse to certain extent,
their robustness remains insufficient when dealing with adversarial attacks.
  Given that semantic consistency between input text and output image is a
fundamental requirement for T2I models, we identify that textual
representations (i.e., prompt embeddings) are likely the primary source of
unsafe generation. To this end, we propose a vision-agnostic safe generation
framework, Embedding Sanitizer (ES), which focuses on erasing inappropriate
concepts from prompt embeddings and uses the sanitized embeddings to guide the
model for safe generation. ES is applied to the output of the text encoder as a
plug-and-play module, enabling seamless integration with different T2I models
as well as other safeguards. In addition, ES's unique scoring mechanism assigns
a score to each token in the prompt to indicate its potential harmfulness, and
dynamically adjusts the sanitization intensity to balance defensive performance
and generation quality. Through extensive evaluation on five prompt benchmarks,
our approach achieves state-of-the-art robustness by sanitizing the source
(prompt embedding) of unsafe generation compared to nine baseline methods. It
significantly outperforms existing safeguards in terms of interpretability and
controllability while maintaining generation quality.


http://arxiv.org/abs/2411.10242
Measuring Non-Adversarial Reproduction of Training Data in Large Language Models. (9%)
Michael Aerni; Javier Rando; Edoardo Debenedetti; Nicholas Carlini; Daphne Ippolito; Florian Tramèr
  Large language models memorize parts of their training data. Memorizing short
snippets and facts is required to answer questions about the world and to be
fluent in any language. But models have also been shown to reproduce long
verbatim sequences of memorized text when prompted by a motivated adversary. In
this work, we investigate an intermediate regime of memorization that we call
non-adversarial reproduction, where we quantify the overlap between model
responses and pretraining data when responding to natural and benign prompts.
For a variety of innocuous prompt categories (e.g., writing a letter or a
tutorial), we show that up to 15% of the text output by popular conversational
language models overlaps with snippets from the Internet. In worst cases, we
find generations where 100% of the content can be found exactly online. For the
same tasks, we find that human-written text has far less overlap with Internet
data. We further study whether prompting strategies can close this reproduction
gap between models and humans. While appropriate prompting can reduce
non-adversarial reproduction on average, we find that mitigating worst-case
reproduction of training data requires stronger defenses -- even for benign
interactions.


http://arxiv.org/abs/2411.10029
Toward Robust and Accurate Adversarial Camouflage Generation against Vehicle Detectors. (1%)
Jiawei Zhou; Linye Lyu; Daojing He; Yu Li
  Adversarial camouflage is a widely used physical attack against vehicle
detectors for its superiority in multi-view attack performance. One promising
approach involves using differentiable neural renderers to facilitate
adversarial camouflage optimization through gradient back-propagation. However,
existing methods often struggle to capture environmental characteristics during
the rendering process or produce adversarial textures that can precisely map to
the target vehicle. Moreover, these approaches neglect diverse weather
conditions, reducing the efficacy of generated camouflage across varying
weather scenarios. To tackle these challenges, we propose a robust and accurate
camouflage generation method, namely RAUCA. The core of RAUCA is a novel neural
rendering component, End-to-End Neural Renderer Plus (E2E-NRP), which can
accurately optimize and project vehicle textures and render images with
environmental characteristics such as lighting and weather. In addition, we
integrate a multi-weather dataset for camouflage generation, leveraging the
E2E-NRP to enhance the attack robustness. Experimental results on six popular
object detectors show that RAUCA-final outperforms existing methods in both
simulation and real-world settings.


http://arxiv.org/abs/2411.10507
RedTest: Towards Measuring Redundancy in Deep Neural Networks Effectively. (1%)
Yao Lu; Peixin Zhang; Jingyi Wang; Lei Ma; Xiaoniu Yang; Qi Xuan
  Deep learning has revolutionized computing in many real-world applications,
arguably due to its remarkable performance and extreme convenience as an
end-to-end solution. However, deep learning models can be costly to train and
to use, especially for those large-scale models, making it necessary to
optimize the original overly complicated models into smaller ones in scenarios
with limited resources such as mobile applications or simply for resource
saving. The key question in such model optimization is, how can we effectively
identify and measure the redundancy in a deep learning model structure. While
several common metrics exist in the popular model optimization techniques to
measure the performance of models after optimization, they are not able to
quantitatively inform the degree of remaining redundancy. To address the
problem, we present a novel testing approach, i.e., RedTest, which proposes a
novel testing metric called Model Structural Redundancy Score (MSRS) to
quantitatively measure the degree of redundancy in a deep learning model
structure. We first show that MSRS is effective in both revealing and assessing
the redundancy issues in many state-of-the-art models, which urgently calls for
model optimization. Then, we utilize MSRS to assist deep learning model
developers in two practical application scenarios: 1) in Neural Architecture
Search, we design a novel redundancy-aware algorithm to guide the search for
the optimal model structure and demonstrate its effectiveness by comparing it
to existing standard NAS practice; 2) in the pruning of large-scale pre-trained
models, we prune the redundant layers of pre-trained models with the guidance
of layer similarity to derive less redundant ones of much smaller size.
Extensive experimental results demonstrate that removing such redundancy has a
negligible effect on the model utility.


http://arxiv.org/abs/2411.09265
BEARD: Benchmarking the Adversarial Robustness for Dataset Distillation. (99%)
Zheng Zhou; Wenquan Feng; Shuchang Lyu; Guangliang Cheng; Xiaowei Huang; Qi Zhao
  Dataset Distillation (DD) is an emerging technique that compresses
large-scale datasets into significantly smaller synthesized datasets while
preserving high test performance and enabling the efficient training of large
models. However, current research primarily focuses on enhancing evaluation
accuracy under limited compression ratios, often overlooking critical security
concerns such as adversarial robustness. A key challenge in evaluating this
robustness lies in the complex interactions between distillation methods, model
architectures, and adversarial attack strategies, which complicate standardized
assessments. To address this, we introduce BEARD, an open and unified benchmark
designed to systematically assess the adversarial robustness of DD methods,
including DM, IDM, and BACON. BEARD encompasses a variety of adversarial
attacks (e.g., FGSM, PGD, C&W) on distilled datasets like CIFAR-10/100 and
TinyImageNet. Utilizing an adversarial game framework, it introduces three key
metrics: Robustness Ratio (RR), Attack Efficiency Ratio (AE), and Comprehensive
Robustness-Efficiency Index (CREI). Our analysis includes unified benchmarks,
various Images Per Class (IPC) settings, and the effects of adversarial
training. Results are available on the BEARD Leaderboard, along with a library
providing model and dataset pools to support reproducible research. Access the
code at BEARD.


http://arxiv.org/abs/2411.09220
Transferable Adversarial Attacks against ASR. (89%)
Xiaoxue Gao; Zexin Li; Yiming Chen; Cong Liu; Haizhou Li
  Given the extensive research and real-world applications of automatic speech
recognition (ASR), ensuring the robustness of ASR models against minor input
perturbations becomes a crucial consideration for maintaining their
effectiveness in real-time scenarios. Previous explorations into ASR model
robustness have predominantly revolved around evaluating accuracy on white-box
settings with full access to ASR models. Nevertheless, full ASR model details
are often not available in real-world applications. Therefore, evaluating the
robustness of black-box ASR models is essential for a comprehensive
understanding of ASR model resilience. In this regard, we thoroughly study the
vulnerability of practical black-box attacks in cutting-edge ASR models and
propose to employ two advanced time-domain-based transferable attacks alongside
our differentiable feature extractor. We also propose a speech-aware gradient
optimization approach (SAGO) for ASR, which forces mistranscription with
minimal impact on human imperceptibility through voice activity detection rule
and a speech-aware gradient-oriented optimizer. Our comprehensive experimental
results reveal performance enhancements compared to baseline approaches across
five models on two databases.


http://arxiv.org/abs/2411.09749
Adversarial Attacks Using Differentiable Rendering: A Survey. (83%)
Matthew Hull; Chao Zhang; Zsolt Kira; Duen Horng Chau
  Differentiable rendering methods have emerged as a promising means for
generating photo-realistic and physically plausible adversarial attacks by
manipulating 3D objects and scenes that can deceive deep neural networks
(DNNs). Recently, differentiable rendering capabilities have evolved
significantly into a diverse landscape of libraries, such as Mitsuba,
PyTorch3D, and methods like Neural Radiance Fields and 3D Gaussian Splatting
for solving inverse rendering problems that share conceptually similar
properties commonly used to attack DNNs, such as back-propagation and
optimization. However, the adversarial machine learning research community has
not yet fully explored or understood such capabilities for generating attacks.
Some key reasons are that researchers often have different attack goals, such
as misclassification or misdetection, and use different tasks to accomplish
these goals by manipulating different representation in a scene, such as the
mesh or texture of an object. This survey adopts a task-oriented unifying
framework that systematically summarizes common tasks, such as manipulating
textures, altering illumination, and modifying 3D meshes to exploit
vulnerabilities in DNNs. Our framework enables easy comparison of existing
works, reveals research gaps and spotlights exciting future research directions
in this rapidly evolving field. Through focusing on how these tasks enable
attacks on various DNNs such as image classification, facial recognition,
object detection, optical flow and depth estimation, our survey helps
researchers and practitioners better understand the vulnerabilities of computer
vision systems against photorealistic adversarial attacks that could threaten
real-world applications.


http://arxiv.org/abs/2411.09259
Jailbreak Attacks and Defenses against Multimodal Generative Models: A Survey. (69%)
Xuannan Liu; Xing Cui; Peipei Li; Zekun Li; Huaibo Huang; Shuhan Xia; Miaoxuan Zhang; Yueying Zou; Ran He
  The rapid evolution of multimodal foundation models has led to significant
advancements in cross-modal understanding and generation across diverse
modalities, including text, images, audio, and video. However, these models
remain susceptible to jailbreak attacks, which can bypass built-in safety
mechanisms and induce the production of potentially harmful content.
Consequently, understanding the methods of jailbreak attacks and existing
defense mechanisms is essential to ensure the safe deployment of multimodal
generative models in real-world scenarios, particularly in security-sensitive
applications. To provide comprehensive insight into this topic, this survey
reviews jailbreak and defense in multimodal generative models. First, given the
generalized lifecycle of multimodal jailbreak, we systematically explore
attacks and corresponding defense strategies across four levels: input,
encoder, generator, and output. Based on this analysis, we present a detailed
taxonomy of attack methods, defense mechanisms, and evaluation frameworks
specific to multimodal generative models. Additionally, we cover a wide range
of input-output configurations, including modalities such as Any-to-Text,
Any-to-Vision, and Any-to-Any within generative systems. Finally, we highlight
current research challenges and propose potential directions for future
research.The open-source repository corresponding to this work can be found at
https://github.com/liuxuannan/Awesome-Multimodal-Jailbreak.


http://arxiv.org/abs/2411.09359
Your Fixed Watermark is Fragile: Towards Semantic-Aware Watermark for EaaS Copyright Protection. (11%)
Zekun Fei; Biao Yi; Jianing Geng; Ruiqi He; Lihai Nie; Zheli Liu
  Embedding-as-a-Service (EaaS) has emerged as a successful business pattern
but faces significant challenges related to various forms of copyright
infringement, including API misuse and different attacks. Various studies have
proposed backdoor-based watermarking schemes to protect the copyright of EaaS
services. In this paper, we reveal that previous watermarking schemes possess
semantic-independent characteristics and propose the Semantic Perturbation
Attack (SPA). Our theoretical and experimental analyses demonstrate that this
semantic-independent nature makes current watermarking schemes vulnerable to
adaptive attacks that exploit semantic perturbations test to bypass watermark
verification. To address this vulnerability, we propose the Semantic Aware
Watermarking (SAW) scheme, a robust defense mechanism designed to resist SPA,
by injecting a watermark that adapts to the text semantics. Extensive
experimental results across multiple datasets demonstrate that the True
Positive Rate (TPR) for detecting watermarked samples under SPA can reach up to
more than 95%, rendering previous watermarks ineffective. Meanwhile, our
watermarking scheme can resist such attack while ensuring the watermark
verification capability. Our code is available at
https://github.com/Zk4-ps/EaaS-Embedding-Watermark.


http://arxiv.org/abs/2411.09373
Are nuclear masks all you need for improved out-of-domain generalisation? A closer look at cancer classification in histopathology. (1%)
Dhananjay Tomar; Alexander Binder; Andreas Kleppe
  Domain generalisation in computational histopathology is challenging because
the images are substantially affected by differences among hospitals due to
factors like fixation and staining of tissue and imaging equipment. We
hypothesise that focusing on nuclei can improve the out-of-domain (OOD)
generalisation in cancer detection. We propose a simple approach to improve OOD
generalisation for cancer detection by focusing on nuclear morphology and
organisation, as these are domain-invariant features critical in cancer
detection. Our approach integrates original images with nuclear segmentation
masks during training, encouraging the model to prioritise nuclei and their
spatial arrangement. Going beyond mere data augmentation, we introduce a
regularisation technique that aligns the representations of masks and original
images. We show, using multiple datasets, that our method improves OOD
generalisation and also leads to increased robustness to image corruptions and
adversarial attacks. The source code is available at
https://github.com/undercutspiky/SFL/


http://arxiv.org/abs/2411.08933
Confidence-aware Denoised Fine-tuning of Off-the-shelf Models for Certified Robustness. (95%)
Suhyeok Jang; Seojin Kim; Jinwoo Shin; Jongheon Jeong
  The remarkable advances in deep learning have led to the emergence of many
off-the-shelf classifiers, e.g., large pre-trained models. However, since they
are typically trained on clean data, they remain vulnerable to adversarial
attacks. Despite this vulnerability, their superior performance and
transferability make off-the-shelf classifiers still valuable in practice,
demanding further work to provide adversarial robustness for them in a post-hoc
manner. A recently proposed method, denoised smoothing, leverages a denoiser
model in front of the classifier to obtain provable robustness without
additional training. However, the denoiser often creates hallucination, i.e.,
images that have lost the semantics of their originally assigned class, leading
to a drop in robustness. Furthermore, its noise-and-denoise procedure
introduces a significant distribution shift from the original distribution,
causing the denoised smoothing framework to achieve sub-optimal robustness. In
this paper, we introduce Fine-Tuning with Confidence-Aware Denoised Image
Selection (FT-CADIS), a novel fine-tuning scheme to enhance the certified
robustness of off-the-shelf classifiers. FT-CADIS is inspired by the
observation that the confidence of off-the-shelf classifiers can effectively
identify hallucinated images during denoised smoothing. Based on this, we
develop a confidence-aware training objective to handle such hallucinated
images and improve the stability of fine-tuning from denoised images. In this
way, the classifier can be fine-tuned using only images that are beneficial for
adversarial robustness. We also find that such a fine-tuning can be done by
updating a small fraction of parameters of the classifier. Extensive
experiments demonstrate that FT-CADIS has established the state-of-the-art
certified robustness among denoised smoothing methods across all
$\ell_2$-adversary radius in various benchmarks.


http://arxiv.org/abs/2411.08460
Trap-MID: Trapdoor-based Defense against Model Inversion Attacks. (81%)
Zhen-Ting Liu; Shang-Tse Chen
  Model Inversion (MI) attacks pose a significant threat to the privacy of Deep
Neural Networks by recovering training data distribution from well-trained
models. While existing defenses often rely on regularization techniques to
reduce information leakage, they remain vulnerable to recent attacks. In this
paper, we propose the Trapdoor-based Model Inversion Defense (Trap-MID) to
mislead MI attacks. A trapdoor is integrated into the model to predict a
specific label when the input is injected with the corresponding trigger.
Consequently, this trapdoor information serves as the "shortcut" for MI
attacks, leading them to extract trapdoor triggers rather than private data. We
provide theoretical insights into the impacts of trapdoor's effectiveness and
naturalness on deceiving MI attacks. In addition, empirical experiments
demonstrate the state-of-the-art defense performance of Trap-MID against
various MI attacks without the requirements for extra data or large
computational overhead. Our source code is publicly available at
https://github.com/ntuaislab/Trap-MID.


http://arxiv.org/abs/2411.08618
Robust Optimal Power Flow Against Adversarial Attacks: A Tri-Level Optimization Approach. (81%)
Saman Mazaheri Khamaneh; Tong Wu
  In power systems, unpredictable events like extreme weather, equipment
failures, and cyberattacks present significant challenges to ensuring safety
and reliability. Ensuring resilience in the face of these uncertainties is
crucial for reliable and efficient operations. This paper presents a tri-level
optimization approach for robust power system operations that effectively
address worst-case attacks. The first stage focuses on optimizing economic
dispatch under normal operating conditions, aiming to minimize generation costs
while maintaining the supply-demand balance. The second stage introduces an
adversarial attack model, identifying worst-case scenarios that maximize the
system's vulnerability by targeting distributed generation (DG). In the third
stage, mitigation strategies are developed using fast-response energy storage
systems (ESS) to minimize disruptions caused by these attacks. By integrating
economic dispatch, vulnerability assessment, and mitigation into a unified
framework, this approach provides a robust solution for enhancing power system
resilience and safety against evolving adversarial threats. The approach is
validated using the IEEE-33 node distribution system to demonstrate its
effectiveness in achieving both cost efficiency and system resilience.


http://arxiv.org/abs/2411.08410
The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defense. (22%)
Yangyang Guo; Fangkai Jiao; Liqiang Nie; Mohan Kankanhalli
  The vulnerability of Vision Large Language Models (VLLMs) to jailbreak
attacks appears as no surprise. However, recent defense mechanisms against
these attacks have reached near-saturation performance on benchmarks, often
with minimal effort. This simultaneous high performance in both attack and
defense presents a perplexing paradox. Resolving it is critical for advancing
the development of trustworthy models. To address this research gap, we first
investigate why VLLMs are prone to these attacks. We then make a key
observation: existing defense mechanisms suffer from an \textbf{over-prudence}
problem, resulting in unexpected abstention even in the presence of benign
inputs. Additionally, we find that the two representative evaluation methods
for jailbreak often exhibit chance agreement. This limitation makes it
potentially misleading when evaluating attack strategies or defense mechanisms.
Beyond these empirical observations, our another contribution in this work is
to repurpose the guardrails of LLMs on the shelf, as an effective alternative
detector prior to VLLM response. We believe these findings offer useful
insights to rethink the foundational development of VLLM safety with respect to
benchmark datasets, evaluation methods, and defense strategies.


http://arxiv.org/abs/2411.08862
LLMStinger: Jailbreaking LLMs using RL fine-tuned LLMs. (8%)
Piyush Jha; Arnav Arora; Vijay Ganesh
  We introduce LLMStinger, a novel approach that leverages Large Language
Models (LLMs) to automatically generate adversarial suffixes for jailbreak
attacks. Unlike traditional methods, which require complex prompt engineering
or white-box access, LLMStinger uses a reinforcement learning (RL) loop to
fine-tune an attacker LLM, generating new suffixes based on existing attacks
for harmful questions from the HarmBench benchmark. Our method significantly
outperforms existing red-teaming approaches (we compared against 15 of the
latest methods), achieving a +57.2% improvement in Attack Success Rate (ASR) on
LLaMA2-7B-chat and a +50.3% ASR increase on Claude 2, both models known for
their extensive safety measures. Additionally, we achieved a 94.97% ASR on
GPT-3.5 and 99.4% on Gemma-2B-it, demonstrating the robustness and adaptability
of LLMStinger across open and closed-source models.


http://arxiv.org/abs/2411.07843
Chain Association-based Attacking and Shielding Natural Language Processing Systems. (99%)
Jiacheng Huang; Long Chen
  Association as a gift enables people do not have to mention something in
completely straightforward words and allows others to understand what they
intend to refer to. In this paper, we propose a chain association-based
adversarial attack against natural language processing systems, utilizing the
comprehension gap between humans and machines. We first generate a chain
association graph for Chinese characters based on the association paradigm for
building search space of potential adversarial examples. Then, we introduce an
discrete particle swarm optimization algorithm to search for the optimal
adversarial examples. We conduct comprehensive experiments and show that
advanced natural language processing models and applications, including large
language models, are vulnerable to our attack, while humans appear good at
understanding the perturbed text. We also explore two methods, including
adversarial training and associative graph-based recovery, to shield systems
from chain association-based attack. Since a few examples that use some
derogatory terms, this paper contains materials that may be offensive or
upsetting to some people.


http://arxiv.org/abs/2411.07850
IAE: Irony-based Adversarial Examples for Sentiment Analysis Systems. (99%)
Xiaoyin Yi; Jiacheng Huang
  Adversarial examples, which are inputs deliberately perturbed with
imperceptible changes to induce model errors, have raised serious concerns for
the reliability and security of deep neural networks (DNNs). While adversarial
attacks have been extensively studied in continuous data domains such as
images, the discrete nature of text presents unique challenges. In this paper,
we propose Irony-based Adversarial Examples (IAE), a method that transforms
straightforward sentences into ironic ones to create adversarial text. This
approach exploits the rhetorical device of irony, where the intended meaning is
opposite to the literal interpretation, requiring a deeper understanding of
context to detect. The IAE method is particularly challenging due to the need
to accurately locate evaluation words, substitute them with appropriate
collocations, and expand the text with suitable ironic elements while
maintaining semantic coherence. Our research makes the following key
contributions: (1) We introduce IAE, a strategy for generating textual
adversarial examples using irony. This method does not rely on pre-existing
irony corpora, making it a versatile tool for creating adversarial text in
various NLP tasks. (2) We demonstrate that the performance of several
state-of-the-art deep learning models on sentiment analysis tasks significantly
deteriorates when subjected to IAE attacks. This finding underscores the
susceptibility of current NLP systems to adversarial manipulation through
irony. (3) We compare the impact of IAE on human judgment versus NLP systems,
revealing that humans are less susceptible to the effects of irony in text.


http://arxiv.org/abs/2411.07559
Zer0-Jack: A Memory-efficient Gradient-based Jailbreaking Method for Black-box Multi-modal Large Language Models. (78%)
Tiejin Chen; Kaishen Wang; Hua Wei
  Jailbreaking methods, which induce Multi-modal Large Language Models (MLLMs)
to output harmful responses, raise significant safety concerns. Among these
methods, gradient-based approaches, which use gradients to generate malicious
prompts, have been widely studied due to their high success rates in white-box
settings, where full access to the model is available. However, these methods
have notable limitations: they require white-box access, which is not always
feasible, and involve high memory usage. To address scenarios where white-box
access is unavailable, attackers often resort to transfer attacks. In transfer
attacks, malicious inputs generated using white-box models are applied to
black-box models, but this typically results in reduced attack performance. To
overcome these challenges, we propose Zer0-Jack, a method that bypasses the
need for white-box access by leveraging zeroth-order optimization. We propose
patch coordinate descent to efficiently generate malicious image inputs to
directly attack black-box MLLMs, which significantly reduces memory usage
further. Through extensive experiments, Zer0-Jack achieves a high attack
success rate across various models, surpassing previous transfer-based methods
and performing comparably with existing white-box jailbreak techniques.
Notably, Zer0-Jack achieves a 95\% attack success rate on MiniGPT-4 with the
Harmful Behaviors Multi-modal Dataset on a black-box setting, demonstrating its
effectiveness. Additionally, we show that Zer0-Jack can directly attack
commercial MLLMs such as GPT-4o. Codes are provided in the supplement.


http://arxiv.org/abs/2411.08248
Deceiving Question-Answering Models: A Hybrid Word-Level Adversarial Approach. (67%)
Jiyao Li; Mingze Ni; Yongshun Gong; Wei Liu
  Deep learning underpins most of the currently advanced natural language
processing (NLP) tasks such as textual classification, neural machine
translation (NMT), abstractive summarization and question-answering (QA).
However, the robustness of the models, particularly QA models, against
adversarial attacks is a critical concern that remains insufficiently explored.
This paper introduces QA-Attack (Question Answering Attack), a novel word-level
adversarial strategy that fools QA models. Our attention-based attack exploits
the customized attention mechanism and deletion ranking strategy to identify
and target specific words within contextual passages. It creates deceptive
inputs by carefully choosing and substituting synonyms, preserving grammatical
integrity while misleading the model to produce incorrect responses. Our
approach demonstrates versatility across various question types, particularly
when dealing with extensive long textual inputs. Extensive experiments on
multiple benchmark datasets demonstrate that QA-Attack successfully deceives
baseline QA models and surpasses existing adversarial techniques regarding
success rate, semantics changes, BLEU score, fluency and grammar error rate.


http://arxiv.org/abs/2411.07597
A Survey on Adversarial Machine Learning for Code Data: Realistic Threats, Countermeasures, and Interpretations. (64%)
Yulong Yang; Haoran Fan; Chenhao Lin; Qian Li; Zhengyu Zhao; Chao Shen; Xiaohong Guan
  Code Language Models (CLMs) have achieved tremendous progress in source code
understanding and generation, leading to a significant increase in research
interests focused on applying CLMs to real-world software engineering tasks in
recent years. However, in realistic scenarios, CLMs are exposed to potential
malicious adversaries, bringing risks to the confidentiality, integrity, and
availability of CLM systems. Despite these risks, a comprehensive analysis of
the security vulnerabilities of CLMs in the extremely adversarial environment
has been lacking. To close this research gap, we categorize existing attack
techniques into three types based on the CIA triad: poisoning attacks
(integrity \& availability infringement), evasion attacks (integrity
infringement), and privacy attacks (confidentiality infringement). We have
collected so far the most comprehensive (79) papers related to adversarial
machine learning for CLM from the research fields of artificial intelligence,
computer security, and software engineering. Our analysis covers each type of
risk, examining threat model categorization, attack techniques, and
countermeasures, while also introducing novel perspectives on eXplainable AI
(XAI) and exploring the interconnections between different risks. Finally, we
identify current challenges and future research opportunities. This study aims
to provide a comprehensive roadmap for both researchers and practitioners and
pave the way towards more reliable CLMs for practical applications.


http://arxiv.org/abs/2411.07691
New Emerged Security and Privacy of Pre-trained Model: a Survey and Outlook. (13%)
Meng Yang; Tianqing Zhu; Chi Liu; WanLei Zhou; Shui Yu; Philip S. Yu
  Thanks to the explosive growth of data and the development of computational
resources, it is possible to build pre-trained models that can achieve
outstanding performance on various tasks, such as neural language processing,
computer vision, and more. Despite their powerful capabilities, pre-trained
models have also sparked attention to the emerging security challenges
associated with their real-world applications. Security and privacy issues,
such as leaking privacy information and generating harmful responses, have
seriously undermined users' confidence in these powerful models. Concerns are
growing as model performance improves dramatically. Researchers are eager to
explore the unique security and privacy issues that have emerged, their
distinguishing factors, and how to defend against them. However, the current
literature lacks a clear taxonomy of emerging attacks and defenses for
pre-trained models, which hinders a high-level and comprehensive understanding
of these questions. To fill the gap, we conduct a systematical survey on the
security risks of pre-trained models, proposing a taxonomy of attack and
defense methods based on the accessibility of pre-trained models' input and
weights in various security test scenarios. This taxonomy categorizes attacks
and defenses into No-Change, Input-Change, and Model-Change approaches. With
the taxonomy analysis, we capture the unique security and privacy issues of
pre-trained models, categorizing and summarizing existing security issues based
on their characteristics. In addition, we offer a timely and comprehensive
review of each category's strengths and limitations. Our survey concludes by
highlighting potential new research opportunities in the security and privacy
of pre-trained models.


http://arxiv.org/abs/2411.08148
Adaptive Meta-Learning for Robust Deepfake Detection: A Multi-Agent Framework to Data Drift and Model Generalization. (1%)
Dinesh Srivasthav P; Badri Narayan Subudhi
  Pioneering advancements in artificial intelligence, especially in genAI, have
enabled significant possibilities for content creation, but also led to
widespread misinformation and false content. The growing sophistication and
realism of deepfakes is raising concerns about privacy invasion, identity
theft, and has societal, business impacts, including reputational damage and
financial loss. Many deepfake detectors have been developed to tackle this
problem. Nevertheless, as for every AI model, the deepfake detectors face the
wrath of lack of considerable generalization to unseen scenarios and
cross-domain deepfakes. Besides, adversarial robustness is another critical
challenge, as detectors drastically underperform to the slightest imperceptible
change. Most state-of-the-art detectors are trained on static datasets and lack
the ability to adapt to emerging deepfake attack trends. These three crucial
challenges though hold paramount importance for reliability in practise,
particularly in the deepfake domain, are also the problems with any other AI
application. This paper proposes an adversarial meta-learning algorithm using
task-specific adaptive sample synthesis and consistency regularization, in a
refinement phase. By focussing on the classifier's strengths and weaknesses, it
boosts both robustness and generalization of the model. Additionally, the paper
introduces a hierarchical multi-agent retrieval-augmented generation workflow
with a sample synthesis module to dynamically adapt the model to new data
trends by generating custom deepfake samples. The paper further presents a
framework integrating the meta-learning algorithm with the hierarchical
multi-agent workflow, offering a holistic solution for enhancing
generalization, robustness, and adaptability. Experimental results demonstrate
the model's consistent performance across various datasets, outperforming the
models in comparison.


http://arxiv.org/abs/2411.06784
Boosting the Targeted Transferability of Adversarial Examples via Salient Region & Weighted Feature Drop. (99%)
Shanjun Xu; Linghui Li; Kaiguo Yuan; Bingyu Li
  Deep neural networks can be vulnerable to adversarially crafted examples,
presenting significant risks to practical applications. A prevalent approach
for adversarial attacks relies on the transferability of adversarial examples,
which are generated from a substitute model and leveraged to attack unknown
black-box models. Despite various proposals aimed at improving transferability,
the success of these attacks in targeted black-box scenarios is often hindered
by the tendency for adversarial examples to overfit to the surrogate models. In
this paper, we introduce a novel framework based on Salient region & Weighted
Feature Drop (SWFD) designed to enhance the targeted transferability of
adversarial examples. Drawing from the observation that examples with higher
transferability exhibit smoother distributions in the deep-layer outputs, we
propose the weighted feature drop mechanism to modulate activation values
according to weights scaled by norm distribution, effectively addressing the
overfitting issue when generating adversarial examples. Additionally, by
leveraging salient region within the image to construct auxiliary images, our
method enables the adversarial example's features to be transferred to the
target category in a model-agnostic manner, thereby enhancing the
transferability. Comprehensive experiments confirm that our approach
outperforms state-of-the-art methods across diverse configurations. On average,
the proposed SWFD raises the attack success rate for normally trained models
and robust models by 16.31% and 7.06% respectively.


http://arxiv.org/abs/2411.06863
Computable Model-Independent Bounds for Adversarial Quantum Machine Learning. (69%)
Bacui Li; Tansu Alpcan; Chandra Thapa; Udaya Parampalli
  By leveraging the principles of quantum mechanics, QML opens doors to novel
approaches in machine learning and offers potential speedup. However, machine
learning models are well-documented to be vulnerable to malicious
manipulations, and this susceptibility extends to the models of QML. This
situation necessitates a thorough understanding of QML's resilience against
adversarial attacks, particularly in an era where quantum computing
capabilities are expanding. In this regard, this paper examines
model-independent bounds on adversarial performance for QML. To the best of our
knowledge, we introduce the first computation of an approximate lower bound for
adversarial error when evaluating model resilience against sophisticated
quantum-based adversarial attacks. Experimental results are compared to the
computed bound, demonstrating the potential of QML models to achieve high
robustness. In the best case, the experimental error is only 10% above the
estimated bound, offering evidence of the inherent robustness of quantum
models. This work not only advances our theoretical understanding of quantum
model resilience but also provides a precise reference bound for the future
development of robust QML algorithms.


http://arxiv.org/abs/2411.07023
The Inherent Adversarial Robustness of Analog In-Memory Computing. (61%)
Corey Lammie; Julian Büchel; Athanasios Vasilopoulos; Manuel Le Gallo; Abu Sebastian
  A key challenge for Deep Neural Network (DNN) algorithms is their
vulnerability to adversarial attacks. Inherently non-deterministic compute
substrates, such as those based on Analog In-Memory Computing (AIMC), have been
speculated to provide significant adversarial robustness when performing DNN
inference. In this paper, we experimentally validate this conjecture for the
first time on an AIMC chip based on Phase Change Memory (PCM) devices. We
demonstrate higher adversarial robustness against different types of
adversarial attacks when implementing an image classification network.
Additional robustness is also observed when performing hardware-in-the-loop
attacks, for which the attacker is assumed to have full access to the hardware.
A careful study of the various noise sources indicate that a combination of
stochastic noise sources (both recurrent and non-recurrent) are responsible for
the adversarial robustness and that their type and magnitude disproportionately
effects this property. Finally, it is demonstrated, via simulations, that when
a much larger transformer network is used to implement a Natural Language
Processing (NLP) task, additional robustness is still observed.


http://arxiv.org/abs/2411.07494
Rapid Response: Mitigating LLM Jailbreaks with a Few Examples. (54%)
Alwin Peng; Julian Michael; Henry Sleight; Ethan Perez; Mrinank Sharma
  As large language models (LLMs) grow more powerful, ensuring their safety
against misuse becomes crucial. While researchers have focused on developing
robust defenses, no method has yet achieved complete invulnerability to
attacks. We propose an alternative approach: instead of seeking perfect
adversarial robustness, we develop rapid response techniques to look to block
whole classes of jailbreaks after observing only a handful of attacks. To study
this setting, we develop RapidResponseBench, a benchmark that measures a
defense's robustness against various jailbreak strategies after adapting to a
few observed examples. We evaluate five rapid response methods, all of which
use jailbreak proliferation, where we automatically generate additional
jailbreaks similar to the examples observed. Our strongest method, which
fine-tunes an input classifier to block proliferated jailbreaks, reduces attack
success rate by a factor greater than 240 on an in-distribution set of
jailbreaks and a factor greater than 15 on an out-of-distribution set, having
observed just one example of each jailbreaking strategy. Moreover, further
studies suggest that the quality of proliferation model and number of
proliferated examples play an key role in the effectiveness of this defense.
Overall, our results highlight the potential of responding rapidly to novel
jailbreaks to limit LLM misuse.


http://arxiv.org/abs/2411.07472
Semi-Truths: A Large-Scale Dataset of AI-Augmented Images for Evaluating Robustness of AI-Generated Image detectors. (1%)
Anisha Pal; Julia Kruk; Mansi Phute; Manognya Bhattaram; Diyi Yang; Duen Horng Chau; Judy Hoffman
  Text-to-image diffusion models have impactful applications in art, design,
and entertainment, yet these technologies also pose significant risks by
enabling the creation and dissemination of misinformation. Although recent
advancements have produced AI-generated image detectors that claim robustness
against various augmentations, their true effectiveness remains uncertain. Do
these detectors reliably identify images with different levels of augmentation?
Are they biased toward specific scenes or data distributions? To investigate,
we introduce SEMI-TRUTHS, featuring 27,600 real images, 223,400 masks, and
1,472,700 AI-augmented images that feature targeted and localized perturbations
produced using diverse augmentation techniques, diffusion models, and data
distributions. Each augmented image is accompanied by metadata for standardized
and targeted evaluation of detector robustness. Our findings suggest that
state-of-the-art detectors exhibit varying sensitivities to the types and
degrees of perturbations, data distributions, and augmentation methods used,
offering new insights into their performance and limitations. The code for the
augmentation and evaluation pipeline is available at
https://github.com/J-Kruk/SemiTruths.


http://arxiv.org/abs/2411.06666
Adversarial Detection with a Dynamically Stable System. (99%)
Xiaowei Long; Jie Lin; Xiangyuan Yang
  Adversarial detection is designed to identify and reject maliciously crafted
adversarial examples(AEs) which are generated to disrupt the classification of
target models.
  Presently, various input transformation-based methods have been developed on
adversarial example detection, which typically rely on empirical experience and
lead to unreliability against new attacks.
  To address this issue, we propose and conduct a Dynamically Stable System
(DSS), which can effectively detect the adversarial examples from normal
examples according to the stability of input examples.
  Particularly, in our paper, the generation of adversarial examples is
considered as the perturbation process of a Lyapunov dynamic system, and we
propose an example stability mechanism, in which a novel control term is added
in adversarial example generation to ensure that the normal examples can
achieve dynamic stability while the adversarial examples cannot achieve the
stability.
  Then, based on the proposed example stability mechanism, a Dynamically Stable
System (DSS) is proposed, which can utilize the disruption and restoration
actions to determine the stability of input examples and detect the adversarial
examples through changes in the stability of the input examples.
  In comparison with existing methods in three benchmark datasets(MNIST,
CIFAR10, and CIFAR100), our evaluation results show that our proposed DSS can
achieve ROC-AUC values of 99.83%, 97.81% and 94.47%, surpassing the
state-of-the-art(SOTA) values of 97.35%, 91.10% and 93.49% in the other 7
methods.


http://arxiv.org/abs/2411.14449
Deferred Backdoor Functionality Attacks on Deep Learning Models. (82%)
Jeongjin Shin; Sangdon Park
  Deep learning models are vulnerable to backdoor attacks, where adversaries
inject malicious functionality during training that activates on trigger inputs
at inference time. Extensive research has focused on developing stealthy
backdoor attacks to evade detection and defense mechanisms. However, these
approaches still have limitations that leave the door open for detection and
mitigation due to their inherent design to cause malicious behavior in the
presence of a trigger. To address this limitation, we introduce Deferred
Backdoor Functionality Activation (DBFA), a new paradigm in backdoor attacks.
Unlike conventional attacks, DBFA initially conceals its backdoor, producing
benign outputs even when triggered. This stealthy behavior allows DBFA to
bypass multiple detection and defense methods, remaining undetected during
initial inspections. The backdoor functionality is strategically activated only
after the model undergoes subsequent updates, such as retraining on benign
data. DBFA attacks exploit the common practice in the life cycle of machine
learning models to perform model updates and fine-tuning after initial
deployment. To implement DBFA attacks, we approach the problem by making the
unlearning of the backdoor fragile, allowing it to be easily cancelled and
subsequently reactivate the backdoor functionality. To achieve this, we propose
a novel two-stage training scheme, called DeferBad. Our extensive experiments
across various fine-tuning scenarios, backdoor attack types, datasets, and
model architectures demonstrate the effectiveness and stealthiness of DeferBad.


http://arxiv.org/abs/2411.06426
SequentialBreak: Large Language Models Can be Fooled by Embedding Jailbreak Prompts into Sequential Prompt Chains. (70%)
Bijoy Ahmed Saiem; MD Sadik Hossain Shanto; Rakib Ahsan; Md Rafi ur Rashid
  As the integration of the Large Language Models (LLMs) into various
applications increases, so does their susceptibility to misuse, raising
significant security concerns. Numerous jailbreak attacks have been proposed to
assess the security defense of LLMs. Current jailbreak attacks mainly rely on
scenario camouflage, prompt obfuscation, prompt optimization, and prompt
iterative optimization to conceal malicious prompts. In particular, sequential
prompt chains in a single query can lead LLMs to focus on certain prompts while
ignoring others, facilitating context manipulation. This paper introduces
SequentialBreak, a novel jailbreak attack that exploits this vulnerability. We
discuss several scenarios, not limited to examples like Question Bank, Dialog
Completion, and Game Environment, where the harmful prompt is embedded within
benign ones that can fool LLMs into generating harmful responses. The distinct
narrative structures of these scenarios show that SequentialBreak is flexible
enough to adapt to various prompt formats beyond those discussed. Extensive
experiments demonstrate that SequentialBreak uses only a single query to
achieve a substantial gain of attack success rate over existing baselines
against both open-source and closed-source models. Through our research, we
highlight the urgent need for more robust and resilient safeguards to enhance
LLM security and prevent potential misuse. All the result files and website
associated with this research are available in this GitHub repository:
https://anonymous.4open.science/r/JailBreakAttack-4F3B/.


http://arxiv.org/abs/2411.07268
Target-driven Attack for Large Language Models. (73%)
Chong Zhang; Mingyu Jin; Dong Shu; Taowen Wang; Dongfang Liu; Xiaobo Jin
  Current large language models (LLM) provide a strong foundation for
large-scale user-oriented natural language tasks. Many users can easily inject
adversarial text or instructions through the user interface, thus causing LLM
model security challenges like the language model not giving the correct
answer. Although there is currently a large amount of research on black-box
attacks, most of these black-box attacks use random and heuristic strategies.
It is unclear how these strategies relate to the success rate of attacks and
thus effectively improve model robustness. To solve this problem, we propose
our target-driven black-box attack method to maximize the KL divergence between
the conditional probabilities of the clean text and the attack text to redefine
the attack's goal. We transform the distance maximization problem into two
convex optimization problems based on the attack goal to solve the attack text
and estimate the covariance. Furthermore, the projected gradient descent
algorithm solves the vector corresponding to the attack text. Our target-driven
black-box attack approach includes two attack strategies: token manipulation
and misinformation attack. Experimental results on multiple Large Language
Models and datasets demonstrate the effectiveness of our attack method.


http://arxiv.org/abs/2411.06146
AI-Compass: A Comprehensive and Effective Multi-module Testing Tool for AI Systems. (33%)
Zhiyu Zhu; Zhibo Jin; Hongsheng Hu; Minhui Xue; Ruoxi Sun; Seyit Camtepe; Praveen Gauravaram; Huaming Chen
  AI systems, in particular with deep learning techniques, have demonstrated
superior performance for various real-world applications. Given the need for
tailored optimization in specific scenarios, as well as the concerns related to
the exploits of subsurface vulnerabilities, a more comprehensive and in-depth
testing AI system becomes a pivotal topic. We have seen the emergence of
testing tools in real-world applications that aim to expand testing
capabilities. However, they often concentrate on ad-hoc tasks, rendering them
unsuitable for simultaneously testing multiple aspects or components.
Furthermore, trustworthiness issues arising from adversarial attacks and the
challenge of interpreting deep learning models pose new challenges for
developing more comprehensive and in-depth AI system testing tools. In this
study, we design and implement a testing tool, \tool, to comprehensively and
effectively evaluate AI systems. The tool extensively assesses multiple
measurements towards adversarial robustness, model interpretability, and
performs neuron analysis. The feasibility of the proposed testing tool is
thoroughly validated across various modalities, including image classification,
object detection, and text classification. Extensive experiments demonstrate
that \tool is the state-of-the-art tool for a comprehensive assessment of the
robustness and trustworthiness of AI systems. Our research sheds light on a
general solution for AI systems testing landscape.


http://arxiv.org/abs/2411.05399
Post-Hoc Robustness Enhancement in Graph Neural Networks with Conditional Random Fields. (41%)
Yassine Abbahaddou; Sofiane Ennadir; Johannes F. Lutzeyer; Fragkiskos D. Malliaros; Michalis Vazirgiannis
  Graph Neural Networks (GNNs), which are nowadays the benchmark approach in
graph representation learning, have been shown to be vulnerable to adversarial
attacks, raising concerns about their real-world applicability. While existing
defense techniques primarily concentrate on the training phase of GNNs,
involving adjustments to message passing architectures or pre-processing
methods, there is a noticeable gap in methods focusing on increasing robustness
during inference. In this context, this study introduces RobustCRF, a post-hoc
approach aiming to enhance the robustness of GNNs at the inference stage. Our
proposed method, founded on statistical relational learning using a Conditional
Random Field, is model-agnostic and does not require prior knowledge about the
underlying model architecture. We validate the efficacy of this approach across
various models, leveraging benchmark node classification datasets.


http://arxiv.org/abs/2411.05345
Reasoning Robustness of LLMs to Adversarial Typographical Errors. (13%)
Esther Gan; Yiran Zhao; Liying Cheng; Yancan Mao; Anirudh Goyal; Kenji Kawaguchi; Min-Yen Kan; Michael Shieh
  Large Language Models (LLMs) have demonstrated impressive capabilities in
reasoning using Chain-of-Thought (CoT) prompting. However, CoT can be biased by
users' instruction. In this work, we study the reasoning robustness of LLMs to
typographical errors, which can naturally occur in users' queries. We design an
Adversarial Typo Attack ($\texttt{ATA}$) algorithm that iteratively samples
typos for words that are important to the query and selects the edit that is
most likely to succeed in attacking. It shows that LLMs are sensitive to
minimal adversarial typographical changes. Notably, with 1 character edit,
Mistral-7B-Instruct's accuracy drops from 43.7% to 38.6% on GSM8K, while with 8
character edits the performance further drops to 19.2%. To extend our
evaluation to larger and closed-source LLMs, we develop the $\texttt{R$^2$ATA}$
benchmark, which assesses models' $\underline{R}$easoning
$\underline{R}$obustness to $\underline{\texttt{ATA}}$. It includes adversarial
typographical questions derived from three widely used reasoning
datasets-GSM8K, BBH, and MMLU-by applying $\texttt{ATA}$ to open-source LLMs.
$\texttt{R$^2$ATA}$ demonstrates remarkable transferability and causes notable
performance drops across multiple super large and closed-source LLMs.


http://arxiv.org/abs/2411.05658
Towards a Re-evaluation of Data Forging Attacks in Practice. (2%)
Mohamed Suliman; Anisa Halimi; Swanand Kadhe; Nathalie Baracaldo; Douglas Leith
  Data forging attacks provide counterfactual proof that a model was trained on
a given dataset, when in fact, it was trained on another. These attacks work by
forging (replacing) mini-batches with ones containing distinct training
examples that produce nearly identical gradients. Data forging appears to break
any potential avenues for data governance, as adversarial model owners may
forge their training set from a dataset that is not compliant to one that is.
Given these serious implications on data auditing and compliance, we critically
analyse data forging from both a practical and theoretical point of view,
finding that a key practical limitation of current attack methods makes them
easily detectable by a verifier; namely that they cannot produce sufficiently
identical gradients. Theoretically, we analyse the question of whether two
distinct mini-batches can produce the same gradient. Generally, we find that
while there may exist an infinite number of distinct mini-batches with
real-valued training examples and labels that produce the same gradient,
finding those that are within the allowed domain e.g. pixel values between
0-255 and one hot labels is a non trivial task. Our results call for the
reevaluation of the strength of existing attacks, and for additional research
into successful data forging, given the serious consequences it may have on
machine learning and privacy.


http://arxiv.org/abs/2411.04533
Neural Fingerprints for Adversarial Attack Detection. (99%)
Haim Fisher; Moni Shahar; Yehezkel S. Resheff
  Deep learning models for image classification have become standard tools in
recent years. A well known vulnerability of these models is their
susceptibility to adversarial examples. These are generated by slightly
altering an image of a certain class in a way that is imperceptible to humans
but causes the model to classify it wrongly as another class. Many algorithms
have been proposed to address this problem, falling generally into one of two
categories: (i) building robust classifiers (ii) directly detecting attacked
images. Despite the good performance of these detectors, we argue that in a
white-box setting, where the attacker knows the configuration and weights of
the network and the detector, they can overcome the detector by running many
examples on a local copy, and sending only those that were not detected to the
actual model. This problem is common in security applications where even a very
good model is not sufficient to ensure safety. In this paper we propose to
overcome this inherent limitation of any static defence with randomization. To
do so, one must generate a very large family of detectors with consistent
performance, and select one or more of them randomly for each input. For the
individual detectors, we suggest the method of neural fingerprints. In the
training phase, for each class we repeatedly sample a tiny random subset of
neurons from certain layers of the network, and if their average is
sufficiently different between clean and attacked images of the focal class
they are considered a fingerprint and added to the detector bank. During test
time, we sample fingerprints from the bank associated with the label predicted
by the model, and detect attacks using a likelihood ratio test. We evaluate our
detectors on ImageNet with different attack methods and model architectures,
and show near-perfect detection with low rates of false detection.


http://arxiv.org/abs/2411.05189
Adversarial Robustness of In-Context Learning in Transformers for Linear Regression. (98%)
Usman Anwar; Oswald Johannes Von; Louis Kirsch; David Krueger; Spencer Frei
  Transformers have demonstrated remarkable in-context learning capabilities
across various domains, including statistical learning tasks. While previous
work has shown that transformers can implement common learning algorithms, the
adversarial robustness of these learned algorithms remains unexplored. This
work investigates the vulnerability of in-context learning in transformers to
\textit{hijacking attacks} focusing on the setting of linear regression tasks.
Hijacking attacks are prompt-manipulation attacks in which the adversary's goal
is to manipulate the prompt to force the transformer to generate a specific
output. We first prove that single-layer linear transformers, known to
implement gradient descent in-context, are non-robust and can be manipulated to
output arbitrary predictions by perturbing a single example in the in-context
training set. While our experiments show these attacks succeed on linear
transformers, we find they do not transfer to more complex transformers with
GPT-2 architectures. Nonetheless, we show that these transformers can be
hijacked using gradient-based adversarial attacks. We then demonstrate that
adversarial training enhances transformers' robustness against hijacking
attacks, even when just applied during finetuning. Additionally, we find that
in some settings, adversarial training against a weaker attack model can lead
to robustness to a stronger attack model. Lastly, we investigate the
transferability of hijacking attacks across transformers of varying scales and
initialization seeds, as well as between transformers and ordinary least
squares (OLS). We find that while attacks transfer effectively between
small-scale transformers, they show poor transferability in other scenarios
(small-to-large scale, large-to-large scale, and between transformers and OLS).


http://arxiv.org/abs/2411.05056
Seeing is Deceiving: Exploitation of Visual Pathways in Multi-Modal Language Models. (97%)
Pete Janowczyk; Linda Laurier; Ave Giulietta; Arlo Octavia; Meade Cleti
  Multi-Modal Language Models (MLLMs) have transformed artificial intelligence
by combining visual and text data, making applications like image captioning,
visual question answering, and multi-modal content creation possible. This
ability to understand and work with complex information has made MLLMs useful
in areas such as healthcare, autonomous systems, and digital content. However,
integrating multiple types of data also creates security risks. Attackers can
manipulate either the visual or text inputs, or both, to make the model produce
unintended or even harmful responses. This paper reviews how visual inputs in
MLLMs can be exploited by various attack strategies. We break down these
attacks into categories: simple visual tweaks and cross-modal manipulations, as
well as advanced strategies like VLATTACK, HADES, and Collaborative Multimodal
Adversarial Attack (Co-Attack). These attacks can mislead even the most robust
models while looking nearly identical to the original visuals, making them hard
to detect. We also discuss the broader security risks, including threats to
privacy and safety in important applications. To counter these risks, we review
current defense methods like the SmoothVLM framework, pixel-wise randomization,
and MirrorCheck, looking at their strengths and limitations. We also discuss
new methods to make MLLMs more secure, including adaptive defenses, better
evaluation tools, and security approaches that protect both visual and text
data. By bringing together recent developments and identifying key areas for
improvement, this review aims to support the creation of more secure and
reliable multi-modal AI systems for real-world use.


http://arxiv.org/abs/2411.04772
Attention Masks Help Adversarial Attacks to Bypass Safety Detectors. (97%)
Yunfan Shi
  Despite recent research advancements in adversarial attack methods, current
approaches against XAI monitors are still discoverable and slower. In this
paper, we present an adaptive framework for attention mask generation to enable
stealthy, explainable and efficient PGD image classification adversarial attack
under XAI monitors. Specifically, we utilize mutation XAI mixture and multitask
self-supervised X-UNet for attention mask generation to guide PGD attack.
Experiments on MNIST (MLP), CIFAR-10 (AlexNet) have shown that our system can
outperform benchmark PGD, Sparsefool and SOTA SINIFGSM in balancing among
stealth, efficiency and explainability which is crucial for effectively fooling
SOTA defense protected classifiers.


http://arxiv.org/abs/2411.04811
Defending Deep Regression Models against Backdoor Attacks. (78%)
Lingyu Du; Yupei Liu; Jinyuan Jia; Guohao Lan
  Deep regression models are used in a wide variety of safety-critical
applications, but are vulnerable to backdoor attacks. Although many defenses
have been proposed for classification models, they are ineffective as they do
not consider the uniqueness of regression models. First, the outputs of
regression models are continuous values instead of discretized labels. Thus,
the potential infected target of a backdoored regression model has infinite
possibilities, which makes it impossible to be determined by existing defenses.
Second, the backdoor behavior of backdoored deep regression models is triggered
by the activation values of all the neurons in the feature space, which makes
it difficult to be detected and mitigated using existing defenses. To resolve
these problems, we propose DRMGuard, the first defense to identify if a deep
regression model in the image domain is backdoored or not. DRMGuard formulates
the optimization problem for reverse engineering based on the unique
output-space and feature-space characteristics of backdoored deep regression
models. We conduct extensive evaluations on two regression tasks and four
datasets. The results show that DRMGuard can consistently defend against
various backdoor attacks. We also generalize four state-of-the-art defenses
designed for classifiers to regression models, and compare DRMGuard with them.
The results show that DRMGuard significantly outperforms all those defenses.


http://arxiv.org/abs/2411.05197
Hardware and Software Platform Inference. (5%)
Cheng Zhang; Hanna Foerster; Robert D. Mullins; Yiren Zhao; Ilia Shumailov
  It is now a common business practice to buy access to large language model
(LLM) inference rather than self-host, because of significant upfront hardware
infrastructure and energy costs. However, as a buyer, there is no mechanism to
verify the authenticity of the advertised service including the serving
hardware platform, e.g. that it is actually being served using an NVIDIA H100.
Furthermore, there are reports suggesting that model providers may deliver
models that differ slightly from the advertised ones, often to make them run on
less expensive hardware. That way, a client pays premium for a capable model
access on more expensive hardware, yet ends up being served by a (potentially
less capable) cheaper model on cheaper hardware. In this paper we introduce
\textit{\textbf{hardware and software platform inference (HSPI)}} -- a method
for identifying the underlying \GPU{} architecture and software stack of a
(black-box) machine learning model solely based on its input-output behavior.
Our method leverages the inherent differences of various \GPU{} architectures
and compilers to distinguish between different \GPU{} types and software
stacks. By analyzing the numerical patterns in the model's outputs, we propose
a classification framework capable of accurately identifying the \GPU{} used
for model inference as well as the underlying software configuration. Our
findings demonstrate the feasibility of inferring \GPU{} type from black-box
models. We evaluate HSPI against models served on different real hardware and
find that in a white-box setting we can distinguish between different \GPU{}s
with between $83.9\%$ and $100\%$ accuracy. Even in a black-box setting we are
able to achieve results that are up to three times higher than random guess
accuracy.


http://arxiv.org/abs/2411.05858
Saliency Assisted Quantization for Neural Networks. (1%)
Elmira Mousa Rezabeyk; Salar Beigzad; Yasin Hamzavi; Mohsen Bagheritabar; Seyedeh Sogol Mirikhoozani
  Deep learning methods have established a significant place in image
classification. While prior research has focused on enhancing final outcomes,
the opaque nature of the decision-making process in these models remains a
concern for experts. Additionally, the deployment of these methods can be
problematic in resource-limited environments. This paper tackles the inherent
black-box nature of these models by providing real-time explanations during the
training phase, compelling the model to concentrate on the most distinctive and
crucial aspects of the input. Furthermore, we employ established quantization
techniques to address resource constraints. To assess the effectiveness of our
approach, we explore how quantization influences the interpretability and
accuracy of Convolutional Neural Networks through a comparative analysis of
saliency maps from standard and quantized models. Quantization is implemented
during the training phase using the Parameterized Clipping Activation method,
with a focus on the MNIST and FashionMNIST benchmark datasets. We evaluated
three bit-width configurations (2-bit, 4-bit, and mixed 4/2-bit) to explore the
trade-off between efficiency and interpretability, with each configuration
designed to highlight varying impacts on saliency map clarity and model
accuracy. The results indicate that while quantization is crucial for
implementing models on resource-limited devices, it necessitates a trade-off
between accuracy and interpretability. Lower bit-widths result in more
pronounced reductions in both metrics, highlighting the necessity of meticulous
quantization parameter selection in applications where model transparency is
paramount. The study underscores the importance of achieving a balance between
efficiency and interpretability in the deployment of neural networks.


http://arxiv.org/abs/2411.04731
MISGUIDE: Security-Aware Attack Analytics for Smart Grid Load Frequency Control. (1%)
Nur Imtiazul Haque; Prabin Mali; Mohammad Zakaria Haider; Mohammad Ashiqur Rahman; Sumit Paudyal
  Incorporating advanced information and communication technologies into smart
grids (SGs) offers substantial operational benefits while increasing
vulnerability to cyber threats like false data injection (FDI) attacks. Current
SG attack analysis tools predominantly employ formal methods or adversarial
machine learning (ML) techniques with rule-based bad data detectors to analyze
the attack space. However, these attack analytics either generate simplistic
attack vectors detectable by the ML-based anomaly detection models (ADMs) or
fail to identify critical attack vectors from complex controller dynamics in a
feasible time. This paper introduces MISGUIDE, a novel defense-aware attack
analytics designed to extract verifiable multi-time slot-based FDI attack
vectors from complex SG load frequency control dynamics and ADMs, utilizing the
Gurobi optimizer. MISGUIDE can identify optimal (maliciously triggering
under/over frequency relays in minimal time) and stealthy attack vectors. Using
real-world load data, we validate the MISGUIDE-identified attack vectors
through real-time hardware-in-the-loop (OPALRT) simulations of the IEEE 39-bus
system.


http://arxiv.org/abs/2411.04376
Game-Theoretic Defenses for Robust Conformal Prediction Against Adversarial Attacks in Medical Imaging. (95%)
Rui Luo; Jie Bao; Zhixin Zhou; Chuangyin Dang
  Adversarial attacks pose significant threats to the reliability and safety of
deep learning models, especially in critical domains such as medical imaging.
This paper introduces a novel framework that integrates conformal prediction
with game-theoretic defensive strategies to enhance model robustness against
both known and unknown adversarial perturbations. We address three primary
research questions: constructing valid and efficient conformal prediction sets
under known attacks (RQ1), ensuring coverage under unknown attacks through
conservative thresholding (RQ2), and determining optimal defensive strategies
within a zero-sum game framework (RQ3). Our methodology involves training
specialized defensive models against specific attack types and employing
maximum and minimum classifiers to aggregate defenses effectively. Extensive
experiments conducted on the MedMNIST datasets, including PathMNIST,
OrganAMNIST, and TissueMNIST, demonstrate that our approach maintains high
coverage guarantees while minimizing prediction set sizes. The game-theoretic
analysis reveals that the optimal defensive strategy often converges to a
singular robust model, outperforming uniform and simple strategies across all
evaluated datasets. This work advances the state-of-the-art in uncertainty
quantification and adversarial robustness, providing a reliable mechanism for
deploying deep learning models in adversarial environments.


http://arxiv.org/abs/2411.03752
Deferred Poisoning: Making the Model More Vulnerable via Hessian Singularization. (86%)
Yuhao He; Jinyu Tian; Xianwei Zheng; Li Dong; Yuanman Li; Leo Yu Zhang; Jiantao Zhou
  Recent studies have shown that deep learning models are very vulnerable to
poisoning attacks. Many defense methods have been proposed to address this
issue. However, traditional poisoning attacks are not as threatening as
commonly believed. This is because they often cause differences in how the
model performs on the training set compared to the validation set. Such
inconsistency can alert defenders that their data has been poisoned, allowing
them to take the necessary defensive actions. In this paper, we introduce a
more threatening type of poisoning attack called the Deferred Poisoning Attack.
This new attack allows the model to function normally during the training and
validation phases but makes it very sensitive to evasion attacks or even
natural noise. We achieve this by ensuring the poisoned model's loss function
has a similar value as a normally trained model at each input sample but with a
large local curvature. A similar model loss ensures that there is no obvious
inconsistency between the training and validation accuracy, demonstrating high
stealthiness. On the other hand, the large curvature implies that a small
perturbation may cause a significant increase in model loss, leading to
substantial performance degradation, which reflects a worse robustness. We
fulfill this purpose by making the model have singular Hessian information at
the optimal point via our proposed Singularization Regularization term. We have
conducted both theoretical and empirical analyses of the proposed method and
validated its effectiveness through experiments on image classification tasks.
Furthermore, we have confirmed the hazards of this form of poisoning attack
under more general scenarios using natural noise, offering a new perspective
for research in the field of security.


http://arxiv.org/abs/2411.03861
FedRISE: Rating Induced Sign Election of Gradients for Byzantine Tolerant Federated Aggregation. (41%)
Joseph Geo Benjamin; Mothilal Asokan; Mohammad Yaqub; Karthik Nandakumar
  One of the most common defense strategies against model poisoning in
federated learning is to employ a robust aggregator mechanism that makes the
training more resilient. Many of the existing Byzantine robust aggregators
provide theoretical guarantees and are empirically effective against certain
categories of attacks. However, we observe that certain high-strength attacks
can subvert the aggregator and collapse the training. In addition, most
aggregators require identifying tolerant settings to converge. Impact of
attacks becomes more pronounced when the number of Byzantines is near-majority,
and becomes harder to evade if the attacker is omniscient with access to data,
honest updates and aggregation methods. Motivated by these observations, we
develop a robust aggregator called FedRISE for cross-silo FL that is consistent
and less susceptible to poisoning updates by an omniscient attacker. The
proposed method explicitly determines the optimal direction of each gradient
through a sign-voting strategy that uses variance-reduced sparse gradients. We
argue that vote weighting based on the cosine similarity of raw gradients is
misleading, and we introduce a sign-based gradient valuation function that
ignores the gradient magnitude. We compare our method against 8 robust
aggregators under 6 poisoning attacks on 3 datasets and architectures. Our
results show that existing robust aggregators collapse for at least some
attacks under severe settings, while FedRISE demonstrates better robustness
because of a stringent gradient inclusion formulation.


http://arxiv.org/abs/2411.03814
MRJ-Agent: An Effective Jailbreak Agent for Multi-Round Dialogue. (11%)
Fengxiang Wang; Ranjie Duan; Peng Xiao; Xiaojun Jia; Shiji Zhao; Cheng Wei; YueFeng Chen; Chongwen Wang; Jialing Tao; Hang Su; Jun Zhu; Hui Xue
  Large Language Models (LLMs) demonstrate outstanding performance in their
reservoir of knowledge and understanding capabilities, but they have also been
shown to be prone to illegal or unethical reactions when subjected to jailbreak
attacks. To ensure their responsible deployment in critical applications, it is
crucial to understand the safety capabilities and vulnerabilities of LLMs.
Previous works mainly focus on jailbreak in single-round dialogue, overlooking
the potential jailbreak risks in multi-round dialogues, which are a vital way
humans interact with and extract information from LLMs. Some studies have
increasingly concentrated on the risks associated with jailbreak in multi-round
dialogues. These efforts typically involve the use of manually crafted
templates or prompt engineering techniques. However, due to the inherent
complexity of multi-round dialogues, their jailbreak performance is limited. To
solve this problem, we propose a novel multi-round dialogue jailbreaking agent,
emphasizing the importance of stealthiness in identifying and mitigating
potential threats to human values posed by LLMs. We propose a risk
decomposition strategy that distributes risks across multiple rounds of queries
and utilizes psychological strategies to enhance attack strength. Extensive
experiments show that our proposed method surpasses other attack methods and
achieves state-of-the-art attack success rate. We will make the corresponding
code and dataset available for future research. The code will be released soon.


http://arxiv.org/abs/2411.04365
Towards Secured Smart Grid 2.0: Exploring Security Threats, Protection Models, and Challenges. (4%)
Lan-Huong Nguyen; Van-Linh Nguyen; Ren-Hung Hwang; Jian-Jhih Kuo; Yu-Wen Chen; Chien-Chung Huang; Ping-I Pan
  Many nations are promoting the green transition in the energy sector to
attain neutral carbon emissions by 2050. Smart Grid 2.0 (SG2) is expected to
explore data-driven analytics and enhance communication technologies to improve
the efficiency and sustainability of distributed renewable energy systems.
These features are beyond smart metering and electric surplus distribution in
conventional smart grids. Given the high dependence on communication networks
to connect distributed microgrids in SG2, potential cascading failures of
connectivity can cause disruption to data synchronization to the remote control
systems. This paper reviews security threats and defense tactics for three
stakeholders: power grid operators, communication network providers, and
consumers. Through the survey, we found that SG2's stakeholders are
particularly vulnerable to substation attacks/vandalism, malware/ransomware
threats, blockchain vulnerabilities and supply chain breakdowns. Furthermore,
incorporating artificial intelligence (AI) into autonomous energy management in
distributed energy resources of SG2 creates new challenges. Accordingly,
adversarial samples and false data injection on electricity reading and
measurement sensors at power plants can fool AI-powered control functions and
cause messy error-checking operations in energy storage, wrong energy
estimation in electric vehicle charging, and even fraudulent transactions in
peer-to-peer energy trading models. Scalable blockchain-based models, physical
unclonable function, interoperable security protocols, and trustworthy AI
models designed for managing distributed microgrids in SG2 are typical
promising protection models for future research.


http://arxiv.org/abs/2411.05034
Mitigating Privacy Risks in LLM Embeddings from Embedding Inversion. (1%)
Tiantian Liu; Hongwei Yao; Tong Wu; Zhan Qin; Feng Lin; Kui Ren; Chun Chen
  Embeddings have become a cornerstone in the functionality of large language
models (LLMs) due to their ability to transform text data into rich, dense
numerical representations that capture semantic and syntactic properties. These
embedding vector databases serve as the long-term memory of LLMs, enabling
efficient handling of a wide range of natural language processing tasks.
However, the surge in popularity of embedding vector databases in LLMs has been
accompanied by significant concerns about privacy leakage. Embedding vector
databases are particularly vulnerable to embedding inversion attacks, where
adversaries can exploit the embeddings to reverse-engineer and extract
sensitive information from the original text data. Existing defense mechanisms
have shown limitations, often struggling to balance security with the
performance of downstream tasks. To address these challenges, we introduce
Eguard, a novel defense mechanism designed to mitigate embedding inversion
attacks. Eguard employs a transformer-based projection network and text mutual
information optimization to safeguard embeddings while preserving the utility
of LLMs. Our approach significantly reduces privacy risks, protecting over 95%
of tokens from inversion while maintaining high performance across downstream
tasks consistent with original embeddings.


http://arxiv.org/abs/2411.02871
Enhancing Adversarial Robustness via Uncertainty-Aware Distributional Adversarial Training. (99%)
Junhao Dong; Xinghua Qu; Z. Jane Wang; Yew-Soon Ong
  Despite remarkable achievements in deep learning across various domains, its
inherent vulnerability to adversarial examples still remains a critical concern
for practical deployment. Adversarial training has emerged as one of the most
effective defensive techniques for improving model robustness against such
malicious inputs. However, existing adversarial training schemes often lead to
limited generalization ability against underlying adversaries with diversity
due to their overreliance on a point-by-point augmentation strategy by mapping
each clean example to its adversarial counterpart during training. In addition,
adversarial examples can induce significant disruptions in the statistical
information w.r.t. the target model, thereby introducing substantial
uncertainty and challenges to modeling the distribution of adversarial
examples. To circumvent these issues, in this paper, we propose a novel
uncertainty-aware distributional adversarial training method, which enforces
adversary modeling by leveraging both the statistical information of
adversarial examples and its corresponding uncertainty estimation, with the
goal of augmenting the diversity of adversaries. Considering the potentially
negative impact induced by aligning adversaries to misclassified clean
examples, we also refine the alignment reference based on the statistical
proximity to clean examples during adversarial training, thereby reframing
adversarial training within a distribution-to-distribution matching framework
interacted between the clean and adversarial domains. Furthermore, we design an
introspective gradient alignment approach via matching input gradients between
these domains without introducing external models. Extensive experiments across
four benchmark datasets and various network architectures demonstrate that our
approach achieves state-of-the-art adversarial robustness and maintains natural
performance.


http://arxiv.org/abs/2411.02974
Region-Guided Attack on the Segment Anything Model (SAM). (99%)
Xiaoliang Liu; Furao Shen; Jian Zhao
  The Segment Anything Model (SAM) is a cornerstone of image segmentation,
demonstrating exceptional performance across various applications, particularly
in autonomous driving and medical imaging, where precise segmentation is
crucial. However, SAM is vulnerable to adversarial attacks that can
significantly impair its functionality through minor input perturbations.
Traditional techniques, such as FGSM and PGD, are often ineffective in
segmentation tasks due to their reliance on global perturbations that overlook
spatial nuances. Recent methods like Attack-SAM-K and UAD have begun to address
these challenges, but they frequently depend on external cues and do not fully
leverage the structural interdependencies within segmentation processes. This
limitation underscores the need for a novel adversarial strategy that exploits
the unique characteristics of segmentation tasks. In response, we introduce the
Region-Guided Attack (RGA), designed specifically for SAM. RGA utilizes a
Region-Guided Map (RGM) to manipulate segmented regions, enabling targeted
perturbations that fragment large segments and expand smaller ones, resulting
in erroneous outputs from SAM. Our experiments demonstrate that RGA achieves
high success rates in both white-box and black-box scenarios, emphasizing the
need for robust defenses against such sophisticated attacks. RGA not only
reveals SAM's vulnerabilities but also lays the groundwork for developing more
resilient defenses against adversarial threats in image segmentation.


http://arxiv.org/abs/2411.02866
Double Whammy: Stealthy Data Manipulation aided Reconstruction Attack on Graph Federated Learning. (91%)
Jinyin Chen; Minying Ma; Haibin Zheng; Qi Xuan
  Recent research has constructed successful graph reconstruction attack (GRA)
on GFL. But these attacks are still challenged in aspects of effectiveness and
stealth. To address the issues, we propose the first Data Manipulation aided
Reconstruction attack on GFL, dubbed as DMan4Rec. The malicious client is born
to manipulate its locally collected data to enhance graph stealing privacy from
benign ones, so as to construct double whammy on GFL. It differs from previous
work in three terms: (1) effectiveness - to fully utilize the sparsity and
feature smoothness of the graph, novel penalty terms are designed adaptive to
diverse similarity functions for connected and unconnected node pairs, as well
as incorporation label smoothing on top of the original cross-entropy loss. (2)
scalability - DMan4Rec is capable of both white-box and black-box attacks via
training a supervised model to infer the posterior probabilities obtained from
limited queries (3) stealthiness - by manipulating the malicious client's node
features, it can maintain the overall graph structure's invariance and conceal
the attack. Comprehensive experiments on four real datasets and three GNN
models demonstrate that DMan4Rec achieves the state-of-the-art (SOTA) attack
performance, e.g., the attack AUC and precision improved by 9.2% and 10.5%
respectively compared with the SOTA baselines. Particularly, DMan4Rec achieves
an AUC score and a precision score of up to 99.59% and 99.56%, respectively in
black-box setting. Nevertheless, the complete overlap of the distribution
graphs supports the stealthiness of the attack. Besides, DMan4Rec still beats
the defensive GFL, which alarms a new threat to GFL.


http://arxiv.org/abs/2411.03022
Flashy Backdoor: Real-world Environment Backdoor Attack on SNNs with DVS Cameras. (75%)
Roberto Riaño; Gorka Abad; Stjepan Picek; Aitor Urbieta
  While security vulnerabilities in traditional Deep Neural Networks (DNNs)
have been extensively studied, the susceptibility of Spiking Neural Networks
(SNNs) to adversarial attacks remains mostly underexplored. Until now, the
mechanisms to inject backdoors into SNN models have been limited to digital
scenarios; thus, we present the first evaluation of backdoor attacks in
real-world environments.
  We begin by assessing the applicability of existing digital backdoor attacks
and identifying their limitations for deployment in physical environments. To
address each of the found limitations, we present three novel backdoor attack
methods on SNNs, i.e., Framed, Strobing, and Flashy Backdoor. We also assess
the effectiveness of traditional backdoor procedures and defenses adapted for
SNNs, such as pruning, fine-tuning, and fine-pruning. The results show that
while these procedures and defenses can mitigate some attacks, they often fail
against stronger methods like Flashy Backdoor or sacrifice too much clean
accuracy, rendering the models unusable.
  Overall, all our methods can achieve up to a 100% Attack Success Rate while
maintaining high clean accuracy in every tested dataset. Additionally, we
evaluate the stealthiness of the triggers with commonly used metrics, finding
them highly stealthy. Thus, we propose new alternatives more suited for
identifying poisoned samples in these scenarios. Our results show that further
research is needed to ensure the security of SNN-based systems against backdoor
attacks and their safe application in real-world scenarios. The code,
experiments, and results are available in our repository.


http://arxiv.org/abs/2411.03231
Formal Logic-guided Robust Federated Learning against Poisoning Attacks. (68%)
Dung Thuy Nguyen; Ziyan An; Taylor T. Johnson; Meiyi Ma; Kevin Leach
  Federated Learning (FL) offers a promising solution to the privacy concerns
associated with centralized Machine Learning (ML) by enabling decentralized,
collaborative learning. However, FL is vulnerable to various security threats,
including poisoning attacks, where adversarial clients manipulate the training
data or model updates to degrade overall model performance. Recognizing this
threat, researchers have focused on developing defense mechanisms to counteract
poisoning attacks in FL systems. However, existing robust FL methods
predominantly focus on computer vision tasks, leaving a gap in addressing the
unique challenges of FL with time series data. In this paper, we present
FLORAL, a defense mechanism designed to mitigate poisoning attacks in federated
learning for time-series tasks, even in scenarios with heterogeneous client
data and a large number of adversarial participants. Unlike traditional
model-centric defenses, FLORAL leverages logical reasoning to evaluate client
trustworthiness by aligning their predictions with global time-series patterns,
rather than relying solely on the similarity of client updates. Our approach
extracts logical reasoning properties from clients, then hierarchically infers
global properties, and uses these to verify client updates. Through formal
logic verification, we assess the robustness of each client contribution,
identifying deviations indicative of adversarial behavior. Experimental results
on two datasets demonstrate the superior performance of our approach compared
to existing baseline methods, highlighting its potential to enhance the
robustness of FL to time series applications. Notably, FLORAL reduced the
prediction error by 93.27\% in the best-case scenario compared to the
second-best baseline. Our code is available at
\url{https://anonymous.4open.science/r/FLORAL-Robust-FTS}.


http://arxiv.org/abs/2411.03279
Oblivious Defense in ML Models: Backdoor Removal without Detection. (15%)
Shafi Goldwasser; Jonathan Shafer; Neekon Vafa; Vinod Vaikuntanathan
  As society grows more reliant on machine learning, ensuring the security of
machine learning systems against sophisticated attacks becomes a pressing
concern. A recent result of Goldwasser, Kim, Vaikuntanathan, and Zamir (2022)
shows that an adversary can plant undetectable backdoors in machine learning
models, allowing the adversary to covertly control the model's behavior.
Backdoors can be planted in such a way that the backdoored machine learning
model is computationally indistinguishable from an honest model without
backdoors.
  In this paper, we present strategies for defending against backdoors in ML
models, even if they are undetectable. The key observation is that it is
sometimes possible to provably mitigate or even remove backdoors without
needing to detect them, using techniques inspired by the notion of random
self-reducibility. This depends on properties of the ground-truth labels
(chosen by nature), and not of the proposed ML model (which may be chosen by an
attacker).
  We give formal definitions for secure backdoor mitigation, and proceed to
show two types of results. First, we show a "global mitigation" technique,
which removes all backdoors from a machine learning model under the assumption
that the ground-truth labels are close to a Fourier-heavy function. Second, we
consider distributions where the ground-truth labels are close to a linear or
polynomial function in $\mathbb{R}^n$. Here, we show "local mitigation"
techniques, which remove backdoors with high probability for every inputs of
interest, and are computationally cheaper than global mitigation. All of our
constructions are black-box, so our techniques work without needing access to
the model's representation (i.e., its code or parameters). Along the way we
prove a simple result for robust mean estimation.


http://arxiv.org/abs/2411.03364
DM4Steal: Diffusion Model For Link Stealing Attack On Graph Neural Networks. (13%)
Jinyin Chen; Haonan Ma; Haibin Zheng
  Graph has become increasingly integral to the advancement of recommendation
systems, particularly with the fast development of graph neural network(GNN).
By exploring the virtue of rich node features and link information, GNN is
designed to provide personalized and accurate suggestions. Meanwhile, the
privacy leakage of GNN in such contexts has also captured special attention.
Prior work has revealed that a malicious user can utilize auxiliary knowledge
to extract sensitive link data of the target graph, integral to recommendation
systems, via the decision made by the target GNN model. This poses a
significant risk to the integrity and confidentiality of data used in
recommendation system. Though important, previous works on GNN's privacy
leakage are still challenged in three aspects, i.e., limited stealing attack
scenarios, sub-optimal attack performance, and adaptation against defense. To
address these issues, we propose a diffusion model based link stealing attack,
named DM4Steal. It differs previous work from three critical aspects. (i)
Generality: aiming at six attack scenarios with limited auxiliary knowledge, we
propose a novel training strategy for diffusion models so that DM4Steal is
transferable to diverse attack scenarios. (ii) Effectiveness: benefiting from
the retention of semantic structure in the diffusion model during the training
process, DM4Steal is capable to learn the precise topology of the target graph
through the GNN decision process. (iii) Adaptation: when GNN is defensive
(e.g., DP, Dropout), DM4Steal relies on the stability that comes from sampling
the score model multiple times to keep performance degradation to a minimum,
thus DM4Steal implements successful adaptive attack on defensive GNN.


http://arxiv.org/abs/2411.03019
FEDLAD: Federated Evaluation of Deep Leakage Attacks and Defenses. (9%)
Isaac Baglin; Xiatian Zhu; Simon Hadfield
  Federated Learning is a privacy preserving decentralized machine learning
paradigm designed to collaboratively train models across multiple clients by
exchanging gradients to the server and keeping private data local.
Nevertheless, recent research has revealed that the security of Federated
Learning is compromised, as private ground truth data can be recovered through
a gradient inversion technique known as Deep Leakage. While these attacks are
crafted with a focus on applications in Federated Learning, they generally are
not evaluated in realistic scenarios. This paper introduces the FEDLAD
Framework (Federated Evaluation of Deep Leakage Attacks and Defenses), a
comprehensive benchmark for evaluating Deep Leakage attacks and defenses within
a realistic Federated context. By implementing a unified benchmark that
encompasses multiple state-of-the-art Deep Leakage techniques and various
defense strategies, our framework facilitates the evaluation and comparison of
the efficacy of these methods across different datasets and training states.
This work highlights a crucial trade-off between privacy and model accuracy in
Federated Learning and aims to advance the understanding of security challenges
in decentralized machine learning systems, stimulate future research, and
enhance reproducibility in evaluating Deep Leakage attacks and defenses.


http://arxiv.org/abs/2411.02833
Lost in Context: The Influence of Context on Feature Attribution Methods for Object Recognition. (3%)
Sayanta Adhikari; Rishav Kumar; Konda Reddy Mopuri; Rajalakshmi Pachamuthu
  Contextual information plays a critical role in object recognition models
within computer vision, where changes in context can significantly affect
accuracy, underscoring models' dependence on contextual cues. This study
investigates how context manipulation influences both model accuracy and
feature attribution, providing insights into the reliance of object recognition
models on contextual information as understood through the lens of feature
attribution methods.
  We employ a range of feature attribution techniques to decipher the reliance
of deep neural networks on context in object recognition tasks. Using the
ImageNet-9 and our curated ImageNet-CS datasets, we conduct experiments to
evaluate the impact of contextual variations, analyzed through feature
attribution methods. Our findings reveal several key insights: (a) Correctly
classified images predominantly emphasize object volume attribution over
context volume attribution. (b) The dependence on context remains relatively
stable across different context modifications, irrespective of classification
accuracy. (c) Context change exerts a more pronounced effect on model
performance than Context perturbations. (d) Surprisingly, context attribution
in `no-information' scenarios is non-trivial. Our research moves beyond
traditional methods by assessing the implications of broad-level modifications
on object recognition, either in the object or its context.


http://arxiv.org/abs/2411.03554
Benchmarking Vision Language Model Unlearning via Fictitious Facial Identity Dataset. (1%)
Yingzi Ma; Jiongxiao Wang; Fei Wang; Siyuan Ma; Jiazhao Li; Xiujun Li; Furong Huang; Lichao Sun; Bo Li; Yejin Choi; Muhao Chen; Chaowei Xiao
  Machine unlearning has emerged as an effective strategy for forgetting
specific information in the training data. However, with the increasing
integration of visual data, privacy concerns in Vision Language Models (VLMs)
remain underexplored. To address this, we introduce Facial Identity Unlearning
Benchmark (FIUBench), a novel VLM unlearning benchmark designed to robustly
evaluate the effectiveness of unlearning algorithms under the Right to be
Forgotten setting. Specifically, we formulate the VLM unlearning task via
constructing the Fictitious Facial Identity VQA dataset and apply a two-stage
evaluation pipeline that is designed to precisely control the sources of
information and their exposure levels. In terms of evaluation, since VLM
supports various forms of ways to ask questions with the same semantic meaning,
we also provide robust evaluation metrics including membership inference
attacks and carefully designed adversarial privacy attacks to evaluate the
performance of algorithms. Through the evaluation of four baseline VLM
unlearning algorithms within FIUBench, we find that all methods remain limited
in their unlearning performance, with significant trade-offs between model
utility and forget quality. Furthermore, our findings also highlight the
importance of privacy attacks for robust evaluations. We hope FIUBench will
drive progress in developing more effective VLM unlearning algorithms.


http://arxiv.org/abs/2411.02809
Query-Efficient Adversarial Attack Against Vertical Federated Graph Learning. (99%)
Jinyin Chen; Wenbo Mu; Luxin Zhang; Guohan Huang; Haibin Zheng; Yao Cheng
  Graph neural network (GNN) has captured wide attention due to its capability
of graph representation learning for graph-structured data. However, the
distributed data silos limit the performance of GNN. Vertical federated
learning (VFL), an emerging technique to process distributed data, successfully
makes GNN possible to handle the distributed graph-structured data. Despite the
prosperous development of vertical federated graph learning (VFGL), the
robustness of VFGL against the adversarial attack has not been explored yet.
Although numerous adversarial attacks against centralized GNNs are proposed,
their attack performance is challenged in the VFGL scenario. To the best of our
knowledge, this is the first work to explore the adversarial attack against
VFGL. A query-efficient hybrid adversarial attack framework is proposed to
significantly improve the centralized adversarial attacks against VFGL, denoted
as NA2, short for Neuron-based Adversarial Attack. Specifically, a malicious
client manipulates its local training data to improve its contribution in a
stealthy fashion. Then a shadow model is established based on the manipulated
data to simulate the behavior of the server model in VFGL. As a result, the
shadow model can improve the attack success rate of various centralized attacks
with a few queries. Extensive experiments on five real-world benchmarks
demonstrate that NA2 improves the performance of the centralized adversarial
attacks against VFGL, achieving state-of-the-art performance even under
potential adaptive defense where the defender knows the attack method.
Additionally, we provide interpretable experiments of the effectiveness of NA2
via sensitive neurons identification and visualization of t-SNE.


http://arxiv.org/abs/2411.02669
Semantic-Aligned Adversarial Evolution Triangle for High-Transferability Vision-Language Attack. (99%)
Xiaojun Jia; Sensen Gao; Qing Guo; Ke Ma; Yihao Huang; Simeng Qin; Yang Liu; Ivor Tsang Fellow; Xiaochun Cao
  Vision-language pre-training (VLP) models excel at interpreting both images
and text but remain vulnerable to multimodal adversarial examples (AEs).
Advancing the generation of transferable AEs, which succeed across unseen
models, is key to developing more robust and practical VLP models. Previous
approaches augment image-text pairs to enhance diversity within the adversarial
example generation process, aiming to improve transferability by expanding the
contrast space of image-text features. However, these methods focus solely on
diversity around the current AEs, yielding limited gains in transferability. To
address this issue, we propose to increase the diversity of AEs by leveraging
the intersection regions along the adversarial trajectory during optimization.
Specifically, we propose sampling from adversarial evolution triangles composed
of clean, historical, and current adversarial examples to enhance adversarial
diversity. We provide a theoretical analysis to demonstrate the effectiveness
of the proposed adversarial evolution triangle. Moreover, we find that
redundant inactive dimensions can dominate similarity calculations, distorting
feature matching and making AEs model-dependent with reduced transferability.
Hence, we propose to generate AEs in the semantic image-text feature contrast
space, which can project the original feature space into a semantic corpus
subspace. The proposed semantic-aligned subspace can reduce the image feature
redundancy, thereby improving adversarial transferability. Extensive
experiments across different datasets and models demonstrate that the proposed
method can effectively improve adversarial transferability and outperform
state-of-the-art adversarial attack methods. The code is released at
https://github.com/jiaxiaojunQAQ/SA-AET.


http://arxiv.org/abs/2411.01889
LiDAttack: Robust Black-box Attack on LiDAR-based Object Detection. (99%)
Jinyin Chen; Danxin Liao; Sheng Xiang; Haibin Zheng
  Since DNN is vulnerable to carefully crafted adversarial examples,
adversarial attack on LiDAR sensors have been extensively studied. We introduce
a robust black-box attack dubbed LiDAttack. It utilizes a genetic algorithm
with a simulated annealing strategy to strictly limit the location and number
of perturbation points, achieving a stealthy and effective attack. And it
simulates scanning deviations, allowing it to adapt to dynamic changes in real
world scenario variations. Extensive experiments are conducted on 3 datasets
(i.e., KITTI, nuScenes, and self-constructed data) with 3 dominant object
detection models (i.e., PointRCNN, PointPillar, and PV-RCNN++). The results
reveal the efficiency of the LiDAttack when targeting a wide range of object
detection models, with an attack success rate (ASR) up to 90%.


http://arxiv.org/abs/2411.02094
Alignment-Based Adversarial Training (ABAT) for Improving the Robustness and Accuracy of EEG-Based BCIs. (91%)
Xiaoqing Chen; Ziwei Wang; Dongrui Wu
  Machine learning has achieved great success in electroencephalogram (EEG)
based brain-computer interfaces (BCIs). Most existing BCI studies focused on
improving the decoding accuracy, with only a few considering the adversarial
security. Although many adversarial defense approaches have been proposed in
other application domains such as computer vision, previous research showed
that their direct extensions to BCIs degrade the classification accuracy on
benign samples. This phenomenon greatly affects the applicability of
adversarial defense approaches to EEG-based BCIs. To mitigate this problem, we
propose alignment-based adversarial training (ABAT), which performs EEG data
alignment before adversarial training. Data alignment aligns EEG trials from
different domains to reduce their distribution discrepancies, and adversarial
training further robustifies the classification boundary. The integration of
data alignment and adversarial training can make the trained EEG classifiers
simultaneously more accurate and more robust. Experiments on five EEG datasets
from two different BCI paradigms (motor imagery classification, and event
related potential recognition), three convolutional neural network classifiers
(EEGNet, ShallowCNN and DeepCNN) and three different experimental settings
(offline within-subject cross-block/-session classification, online
cross-session classification, and pre-trained classifiers) demonstrated its
effectiveness. It is very intriguing that adversarial attacks, which are
usually used to damage BCI systems, can be used in ABAT to simultaneously
improve the model accuracy and robustness.


http://arxiv.org/abs/2411.02391
Attacking Vision-Language Computer Agents via Pop-ups. (9%)
Yanzhe Zhang; Tao Yu; Diyi Yang
  Autonomous agents powered by large vision and language models (VLM) have
demonstrated significant potential in completing daily computer tasks, such as
browsing the web to book travel and operating desktop software, which requires
agents to understand these interfaces. Despite such visual inputs becoming more
integrated into agentic applications, what types of risks and attacks exist
around them still remain unclear. In this work, we demonstrate that VLM agents
can be easily attacked by a set of carefully designed adversarial pop-ups,
which human users would typically recognize and ignore. This distraction leads
agents to click these pop-ups instead of performing the tasks as usual.
Integrating these pop-ups into existing agent testing environments like OSWorld
and VisualWebArena leads to an attack success rate (the frequency of the agent
clicking the pop-ups) of 86% on average and decreases the task success rate by
47%. Basic defense techniques such as asking the agent to ignore pop-ups or
including an advertisement notice, are ineffective against the attack.


http://arxiv.org/abs/2411.02785
Stochastic Monkeys at Play: Random Augmentations Cheaply Break LLM Safety Alignment. (2%)
Jason Vega; Junsheng Huang; Gaokai Zhang; Hangoo Kang; Minjia Zhang; Gagandeep Singh
  Safety alignment of Large Language Models (LLMs) has recently become a
critical objective of model developers. In response, a growing body of work has
been investigating how safety alignment can be bypassed through various
jailbreaking methods, such as adversarial attacks. However, these jailbreak
methods can be rather costly or involve a non-trivial amount of creativity and
effort, introducing the assumption that malicious users are high-resource or
sophisticated. In this paper, we study how simple random augmentations to the
input prompt affect safety alignment effectiveness in state-of-the-art LLMs,
such as Llama 3 and Qwen 2. We perform an in-depth evaluation of 17 different
models and investigate the intersection of safety under random augmentations
with multiple dimensions: augmentation type, model size, quantization,
fine-tuning-based defenses, and decoding strategies (e.g., sampling
temperature). We show that low-resource and unsophisticated attackers, i.e.
$\textit{stochastic monkeys}$, can significantly improve their chances of
bypassing alignment with just 25 random augmentations per prompt.


http://arxiv.org/abs/2411.02603
FactTest: Factuality Testing in Large Language Models with Statistical Guarantees. (1%)
Fan Nie; Xiaotian Hou; Shuhang Lin; James Zou; Huaxiu Yao; Linjun Zhang
  The propensity of Large Language Models (LLMs) to generate hallucinations and
non-factual content undermines their reliability in high-stakes domains, where
rigorous control over Type I errors (the conditional probability of incorrectly
classifying hallucinations as truthful content) is essential. Despite its
importance, formal verification of LLM factuality with such guarantees remains
largely unexplored. In this paper, we introduce FactTest, a novel framework
that statistically assesses whether an LLM can confidently provide correct
answers to given questions with high-probability correctness guarantees. We
formulate factuality testing as hypothesis testing problem to enforce an upper
bound of Type I errors at user-specified significance levels. Notably, we prove
that our framework also ensures strong Type II error control under mild
conditions and can be extended to maintain its effectiveness when covariate
shifts exist. %These analyses are amenable to the principled NP framework. Our
approach is distribution-free and works for any number of human-annotated
samples. It is model-agnostic and applies to any black-box or white-box LM.
Extensive experiments on question-answering (QA) and multiple-choice benchmarks
demonstrate that \approach effectively detects hallucinations and improves the
model's ability to abstain from answering unknown questions, leading to an over
40% accuracy improvement.


http://arxiv.org/abs/2411.02099
Differentially Private Integrated Decision Gradients (IDG-DP) for Radar-based Human Activity Recognition. (1%)
Idris Zakariyya; Linda Tran; Kaushik Bhargav Sivangi; Paul Henderson; Fani Deligianni
  Human motion analysis offers significant potential for healthcare monitoring
and early detection of diseases. The advent of radar-based sensing systems has
captured the spotlight for they are able to operate without physical contact
and they can integrate with pre-existing Wi-Fi networks. They are also seen as
less privacy-invasive compared to camera-based systems. However, recent
research has shown high accuracy in recognizing subjects or gender from radar
gait patterns, raising privacy concerns. This study addresses these issues by
investigating privacy vulnerabilities in radar-based Human Activity Recognition
(HAR) systems and proposing a novel method for privacy preservation using
Differential Privacy (DP) driven by attributions derived with Integrated
Decision Gradient (IDG) algorithm. We investigate Black-box Membership
Inference Attack (MIA) Models in HAR settings across various levels of
attacker-accessible information. We extensively evaluated the effectiveness of
the proposed IDG-DP method by designing a CNN-based HAR model and rigorously
assessing its resilience against MIAs. Experimental results demonstrate the
potential of IDG-DP in mitigating privacy attacks while maintaining utility
across all settings, particularly excelling against label-only and shadow model
black-box MIA attacks. This work represents a crucial step towards balancing
the need for effective radar-based HAR with robust privacy protection in
healthcare environments.


http://arxiv.org/abs/2411.03348
Undermining Image and Text Classification Algorithms Using Adversarial Attacks. (98%)
Langalibalele Lunga; Suhas Sreehari
  Machine learning models are prone to adversarial attacks, where inputs can be
manipulated in order to cause misclassifications. While previous research has
focused on techniques like Generative Adversarial Networks (GANs), there's
limited exploration of GANs and Synthetic Minority Oversampling Technique
(SMOTE) in text and image classification models to perform adversarial attacks.
Our study addresses this gap by training various machine learning models and
using GANs and SMOTE to generate additional data points aimed at attacking text
classification models. Furthermore, we extend our investigation to face
recognition models, training a Convolutional Neural Network(CNN) and subjecting
it to adversarial attacks with fast gradient sign perturbations on key features
identified by GradCAM, a technique used to highlight key image characteristics
CNNs use in classification. Our experiments reveal a significant vulnerability
in classification models. Specifically, we observe a 20 % decrease in accuracy
for the top-performing text classification models post-attack, along with a 30
% decrease in facial recognition accuracy. This highlights the susceptibility
of these models to manipulation of input data. Adversarial attacks not only
compromise the security but also undermine the reliability of machine learning
systems. By showcasing the impact of adversarial attacks on both text
classification and face recognition models, our study underscores the urgent
need for develop robust defenses against such vulnerabilities.


http://arxiv.org/abs/2411.01565
SQL Injection Jailbreak: a structural disaster of large language models. (78%)
Jiawei Zhao; Kejiang Chen; Weiming Zhang; Nenghai Yu
  In recent years, the rapid development of large language models (LLMs) has
brought new vitality to the various domains and generated substantial social
and economic benefits. However, the swift advancement of LLMs has introduced
new security vulnerabilities. Jailbreak, a form of attack that induces LLMs to
output harmful content through carefully crafted prompts, poses a challenge to
the safe and trustworthy development of LLMs. Previous jailbreak attack methods
primarily exploited the internal capabilities of the model. Among them, one
category leverages the model's implicit capabilities for jailbreak attacks,
where the attacker is unaware of the exact reasons for the attack's success.
The other category utilizes the model's explicit capabilities for jailbreak
attacks, where the attacker understands the reasons for the attack's success.
For example, these attacks exploit the model's abilities in coding, contextual
learning, or understanding ASCII characters. However, these earlier jailbreak
attacks have certain limitations, as they only exploit the inherent
capabilities of the model. In this paper, we propose a novel jailbreak method,
SQL Injection Jailbreak (SIJ), which utilizes the construction of input prompts
by LLMs to inject jailbreak information into user prompts, enabling successful
jailbreak of the LLMs. Our SIJ method achieves nearly 100\% attack success
rates on five well-known open-source LLMs in the context of AdvBench, while
incurring lower time costs compared to previous methods. More importantly, SIJ
reveals a new vulnerability in LLMs that urgently needs to be addressed. To
this end, we propose a defense method called Self-Reminder-Key and demonstrate
its effectiveness through experiments. Our code is available at
\href{https://github.com/weiyezhimeng/SQL-Injection-Jailbreak}{https://github.com/weiyezhimeng/SQL-Injection-Jailbreak}.


http://arxiv.org/abs/2411.01748
Rotation Perturbation Robustness in Point Cloud Analysis: A Perspective of Manifold Distillation. (2%)
Xinyu Xu; Huazhen Liu; Feiming Wei; Huilin Xiong; Wenxian Yu; Tao Zhang
  Point cloud is often regarded as a discrete sampling of Riemannian manifold
and plays a pivotal role in the 3D image interpretation. Particularly, rotation
perturbation, an unexpected small change in rotation caused by various factors
(like equipment offset, system instability, measurement errors and so on), can
easily lead to the inferior results in point cloud learning tasks. However,
classical point cloud learning methods are sensitive to rotation perturbation,
and the existing networks with rotation robustness also have much room for
improvements in terms of performance and noise tolerance. Given these, this
paper remodels the point cloud from the perspective of manifold as well as
designs a manifold distillation method to achieve the robustness of rotation
perturbation without any coordinate transformation. In brief, during the
training phase, we introduce a teacher network to learn the rotation robustness
information and transfer this information to the student network through online
distillation. In the inference phase, the student network directly utilizes the
original 3D coordinate information to achieve the robustness of rotation
perturbation. Experiments carried out on four different datasets verify the
effectiveness of our method. Averagely, on the Modelnet40 and ScanobjectNN
classification datasets with random rotation perturbations, our classification
accuracy has respectively improved by 4.92% and 4.41%, compared to popular
rotation-robust networks; on the ShapeNet and S3DIS segmentation datasets,
compared to the rotation-robust networks, the improvements of mIoU are 7.36%
and 4.82%, respectively. Besides, from the experimental results, the proposed
algorithm also shows excellent performance in resisting noise and outliers.


http://arxiv.org/abs/2412.06788
Poison Attacks and Adversarial Prompts Against an Informed University Virtual Assistant. (1%)
Ivan A. Fernandez; Subash Neupane; Sudip Mittal; Shahram Rahimi
  Recent research has shown that large language models (LLMs) are particularly
vulnerable to adversarial attacks. Since the release of ChatGPT, various
industries are adopting LLM-based chatbots and virtual assistants in their data
workflows. The rapid development pace of AI-based systems is being driven by
the potential of Generative AI (GenAI) to assist humans in decision making. The
immense optimism behind GenAI often overshadows the adversarial risks
associated with these technologies. A threat actor can use security gaps, poor
safeguards, and limited data governance to carry out attacks that grant
unauthorized access to the system and its data. As a proof-of-concept, we
assess the performance of BarkPlug, the Mississippi State University chatbot,
against data poison attacks from a red team perspective.


http://arxiv.org/abs/2411.01779
TabSec: A Collaborative Framework for Novel Insider Threat Detection. (1%)
Zilin Huang; Xiangyan Tang; Hongyu Li; Xinyi Cao; Jieren Cheng
  In the era of the Internet of Things (IoT) and data sharing, users frequently
upload their personal information to enterprise databases to enjoy enhanced
service experiences provided by various online services. However, the
widespread presence of system vulnerabilities, remote network intrusions, and
insider threats significantly increases the exposure of private enterprise data
on the internet. If such data is stolen or leaked by attackers, it can result
in severe asset losses and business operation disruptions. To address these
challenges, this paper proposes a novel threat detection framework, TabITD.
This framework integrates Intrusion Detection Systems (IDS) with User and
Entity Behavior Analytics (UEBA) strategies to form a collaborative detection
system that bridges the gaps in existing systems' capabilities. It effectively
addresses the blurred boundaries between external and insider threats caused by
the diversification of attack methods, thereby enhancing the model's learning
ability and overall detection performance. Moreover, the proposed method
leverages the TabNet architecture, which employs a sparse attention feature
selection mechanism that allows TabNet to select the most relevant features at
each decision step, thereby improving the detection of rare-class attacks. We
evaluated our proposed solution on two different datasets, achieving average
accuracies of 96.71% and 97.25%, respectively. The results demonstrate that
this approach can effectively detect malicious behaviors such as masquerade
attacks and external threats, significantly enhancing network security defenses
and the efficiency of network attack detection.


http://arxiv.org/abs/2411.01777
Learning predictable and robust neural representations by straightening image sequences. (1%)
Xueyan Niu; Cristina Savin; Eero P. Simoncelli
  Prediction is a fundamental capability of all living organisms, and has been
proposed as an objective for learning sensory representations. Recent work
demonstrates that in primate visual systems, prediction is facilitated by
neural representations that follow straighter temporal trajectories than their
initial photoreceptor encoding, which allows for prediction by linear
extrapolation. Inspired by these experimental findings, we develop a
self-supervised learning (SSL) objective that explicitly quantifies and
promotes straightening. We demonstrate the power of this objective in training
deep feedforward neural networks on smoothly-rendered synthetic image sequences
that mimic commonly-occurring properties of natural videos. The learned model
contains neural embeddings that are predictive, but also factorize the
geometric, photometric, and semantic attributes of objects. The representations
also prove more robust to noise and adversarial attacks compared to previous
SSL methods that optimize for invariance to random augmentations. Moreover,
these beneficial properties can be transferred to other training procedures by
using the straightening objective as a regularizer, suggesting a broader
utility for straightening as a principle for robust unsupervised learning.


http://arxiv.org/abs/2411.01222
$B^4$: A Black-Box Scrubbing Attack on LLM Watermarks. (75%)
Baizhou Huang; Xiao Pu; Xiaojun Wan
  Watermarking has emerged as a prominent technique for LLM-generated content
detection by embedding imperceptible patterns. Despite supreme performance, its
robustness against adversarial attacks remains underexplored. Previous work
typically considers a grey-box attack setting, where the specific type of
watermark is already known. Some even necessitates knowledge about
hyperparameters of the watermarking method. Such prerequisites are unattainable
in real-world scenarios. Targeting at a more realistic black-box threat model
with fewer assumptions, we here propose $\mathcal{B}^4$, a black-box scrubbing
attack on watermarks. Specifically, we formulate the watermark scrubbing attack
as a constrained optimization problem by capturing its objectives with two
distributions, a Watermark Distribution and a Fidelity Distribution. This
optimization problem can be approximately solved using two proxy distributions.
Experimental results across 12 different settings demonstrate the superior
performance of $\mathcal{B}^4$ compared with other baselines.


http://arxiv.org/abs/2411.03343
What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attacks. (1%)
Nathalie Maria Kirch; Severin Field; Stephen Casper
  While `jailbreaks' have been central to research on the safety and
reliability of LLMs (large language models), the underlying mechanisms behind
these attacks are not well understood. Some prior works have used linear
methods to analyze jailbreak prompts or model refusal. Here, however, we
compare linear and nonlinear methods to study the features in prompts that
contribute to successful jailbreaks. We do this by probing for jailbreak
success based only on the portions of the latent representations corresponding
to prompt tokens. First, we introduce a dataset of 10,800 jailbreak attempts
from 35 attack methods. We then show that different jailbreaking methods work
via different nonlinear features in prompts. Specifically, we find that while
probes can distinguish between successful and unsuccessful jailbreaking prompts
with a high degree of accuracy, they often transfer poorly to held-out attack
methods. We also show that nonlinear probes can be used to mechanistically
jailbreak the LLM by guiding the design of adversarial latent perturbations.
These mechanistic jailbreaks are able to jailbreak Gemma-7B-IT more reliably
than 34 of the 35 techniques that it was trained on. Ultimately, our results
suggest that jailbreaks cannot be thoroughly understood in terms of universal
or linear prompt features alone.


http://arxiv.org/abs/2411.00898
Replace-then-Perturb: Targeted Adversarial Attacks With Visual Reasoning for Vision-Language Models. (99%)
Jonggyu Jang; Hyeonsu Lyu; Jungyeon Koh; Hyun Jong Yang
  The conventional targeted adversarial attacks add a small perturbation to an
image to make neural network models estimate the image as a predefined target
class, even if it is not the correct target class. Recently, for
visual-language models (VLMs), the focus of targeted adversarial attacks is to
generate a perturbation that makes VLMs answer intended target text outputs.
For example, they aim to make a small perturbation on an image to make VLMs'
answers change from "there is an apple" to "there is a baseball." However,
answering just intended text outputs is insufficient for tricky questions like
"if there is a baseball, tell me what is below it." This is because the target
of the adversarial attacks does not consider the overall integrity of the
original image, thereby leading to a lack of visual reasoning. In this work, we
focus on generating targeted adversarial examples with visual reasoning against
VLMs. To this end, we propose 1) a novel adversarial attack procedure --
namely, Replace-then-Perturb and 2) a contrastive learning-based adversarial
loss -- namely, Contrastive-Adv. In Replace-then-Perturb, we first leverage a
text-guided segmentation model to find the target object in the image. Then, we
get rid of the target object and inpaint the empty space with the desired
prompt. By doing this, we can generate a target image corresponding to the
desired prompt, while maintaining the overall integrity of the original image.
Furthermore, in Contrastive-Adv, we design a novel loss function to obtain
better adversarial examples. Our extensive benchmark results demonstrate that
Replace-then-Perturb and Contrastive-Adv outperform the baseline adversarial
attack algorithms. We note that the source code to reproduce the results will
be available.


http://arxiv.org/abs/2411.00459
Defense Against Prompt Injection Attack by Leveraging Attack Techniques. (81%)
Yulin Chen; Haoran Li; Zihao Zheng; Yangqiu Song; Dekai Wu; Bryan Hooi
  With the advancement of technology, large language models (LLMs) have
achieved remarkable performance across various natural language processing
(NLP) tasks, powering LLM-integrated applications like Microsoft Copilot.
However, as LLMs continue to evolve, new vulnerabilities, especially prompt
injection attacks arise. These attacks trick LLMs into deviating from the
original input instructions and executing the attacker's instructions injected
in data content, such as retrieved results. Recent attack methods leverage
LLMs' instruction-following abilities and their inabilities to distinguish
instructions injected in the data content, and achieve a high attack success
rate (ASR). When comparing the attack and defense methods, we interestingly
find that they share similar design goals, of inducing the model to ignore
unwanted instructions and instead to execute wanted instructions. Therefore, we
raise an intuitive question: Could these attack techniques be utilized for
defensive purposes? In this paper, we invert the intention of prompt injection
methods to develop novel defense methods based on previous training-free attack
methods, by repeating the attack process but with the original input
instruction rather than the injected instruction. Our comprehensive experiments
demonstrate that our defense techniques outperform existing training-free
defense approaches, achieving state-of-the-art results.


http://arxiv.org/abs/2411.00899
Certified Robustness for Deep Equilibrium Models via Serialized Random Smoothing. (68%)
Weizhi Gao; Zhichao Hou; Han Xu; Xiaorui Liu
  Implicit models such as Deep Equilibrium Models (DEQs) have emerged as
promising alternative approaches for building deep neural networks. Their
certified robustness has gained increasing research attention due to security
concerns. Existing certified defenses for DEQs employing deterministic
certification methods such as interval bound propagation and Lipschitz-bounds
can not certify on large-scale datasets. Besides, they are also restricted to
specific forms of DEQs. In this paper, we provide the first randomized
smoothing certified defense for DEQs to solve these limitations. Our study
reveals that simply applying randomized smoothing to certify DEQs provides
certified robustness generalized to large-scale datasets but incurs extremely
expensive computation costs. To reduce computational redundancy, we propose a
novel Serialized Randomized Smoothing (SRS) approach that leverages historical
information. Additionally, we derive a new certified radius estimation for SRS
to theoretically ensure the correctness of our algorithm. Extensive experiments
and ablation studies on image recognition demonstrate that our algorithm can
significantly accelerate the certification of DEQs by up to 7x almost without
sacrificing the certified accuracy. Our code is available at
https://github.com/WeizhiGao/Serialized-Randomized-Smoothing.


http://arxiv.org/abs/2411.00348
Attention Tracker: Detecting Prompt Injection Attacks in LLMs. (26%)
Kuo-Han Hung; Ching-Yun Ko; Ambrish Rawat; I-Hsin Chung; Winston H. Hsu; Pin-Yu Chen
  Large Language Models (LLMs) have revolutionized various domains but remain
vulnerable to prompt injection attacks, where malicious inputs manipulate the
model into ignoring original instructions and executing designated action. In
this paper, we investigate the underlying mechanisms of these attacks by
analyzing the attention patterns within LLMs. We introduce the concept of the
distraction effect, where specific attention heads, termed important heads,
shift focus from the original instruction to the injected instruction. Building
on this discovery, we propose Attention Tracker, a training-free detection
method that tracks attention patterns on instruction to detect prompt injection
attacks without the need for additional LLM inference. Our method generalizes
effectively across diverse models, datasets, and attack types, showing an AUROC
improvement of up to 10.0% over existing methods, and performs well even on
small LLMs. We demonstrate the robustness of our approach through extensive
evaluations and provide insights into safeguarding LLM-integrated systems from
prompt injection vulnerabilities.


http://arxiv.org/abs/2411.01077
Emoji Attack: A Method for Misleading Judge LLMs in Safety Risk Detection. (22%)
Zhipeng Wei; Yuqi Liu; N. Benjamin Erichson
  Jailbreaking attacks show how Large Language Models (LLMs) can be tricked
into generating harmful outputs using malicious prompts. To prevent these
attacks, other LLMs are often used as judges to evaluate the harmfulness of the
generated content. However, relying on LLMs as judges can introduce biases into
the detection process, which in turn compromises the effectiveness of the
evaluation. In this paper, we show that Judge LLMs, like other LLMs, are also
affected by token segmentation bias. This bias occurs when tokens are split
into smaller sub-tokens, altering their embeddings. This makes it harder for
the model to detect harmful content. Specifically, this bias can cause
sub-tokens to differ significantly from the original token in the embedding
space, leading to incorrect "safe" predictions for harmful content. To exploit
this bias in Judge LLMs, we introduce the Emoji Attack -- a method that places
emojis within tokens to increase the embedding differences between sub-tokens
and their originals. These emojis create new tokens that further distort the
token embeddings, exacerbating the bias. To counter the Emoji Attack, we design
prompts that help LLMs filter out unusual characters. However, this defense can
still be bypassed by using a mix of emojis and other characters. The Emoji
Attack can also be combined with existing jailbreaking prompts using few-shot
learning, which enables LLMs to generate harmful responses with emojis. These
responses are often mistakenly labeled as "safe" by Judge LLMs, allowing the
attack to slip through. Our experiments with six state-of-the-art Judge LLMs
show that the Emoji Attack allows 25\% of harmful responses to bypass detection
by Llama Guard and Llama Guard 2, and up to 75\% by ShieldLM. These results
highlight the need for stronger Judge LLMs to address this vulnerability.


http://arxiv.org/abs/2411.00519
Outlier-Oriented Poisoning Attack: A Grey-box Approach to Disturb Decision Boundaries by Perturbing Outliers in Multiclass Learning. (13%)
Anum Paracha; Junaid Arshad; Mohamed Ben Farah; Khalid Ismail
  Poisoning attacks are a primary threat to machine learning models, aiming to
compromise their performance and reliability by manipulating training datasets.
This paper introduces a novel attack - Outlier-Oriented Poisoning (OOP) attack,
which manipulates labels of most distanced samples from the decision
boundaries. The paper also investigates the adverse impact of such attacks on
different machine learning algorithms within a multiclass classification
scenario, analyzing their variance and correlation between different poisoning
levels and performance degradation. To ascertain the severity of the OOP attack
for different degrees (5% - 25%) of poisoning, we analyzed variance, accuracy,
precision, recall, f1-score, and false positive rate for chosen ML
models.Benchmarking our OOP attack, we have analyzed key characteristics of
multiclass machine learning algorithms and their sensitivity to poisoning
attacks. Our experimentation used three publicly available datasets: IRIS,
MNIST, and ISIC. Our analysis shows that KNN and GNB are the most affected
algorithms with a decrease in accuracy of 22.81% and 56.07% while increasing
false positive rate to 17.14% and 40.45% for IRIS dataset with 15% poisoning.
Further, Decision Trees and Random Forest are the most resilient algorithms
with the least accuracy disruption of 12.28% and 17.52% with 15% poisoning of
the IRIS dataset. We have also analyzed the correlation between number of
dataset classes and the performance degradation of models. Our analysis
highlighted that number of classes are inversely proportional to the
performance degradation, specifically the decrease in accuracy of the models,
which is normalized with increasing number of classes. Further, our analysis
identified that imbalanced dataset distribution can aggravate the impact of
poisoning for machine learning models


http://arxiv.org/abs/2411.01040
Identify Backdoored Model in Federated Learning via Individual Unlearning. (5%)
Jiahao Xu; Zikai Zhang; Rui Hu
  Backdoor attacks present a significant threat to the robustness of Federated
Learning (FL) due to their stealth and effectiveness. They maintain both the
main task of the FL system and the backdoor task simultaneously, causing
malicious models to appear statistically similar to benign ones, which enables
them to evade detection by existing defense methods. We find that malicious
parameters in backdoored models are inactive on the main task, resulting in a
significantly large empirical loss during the machine unlearning process on
clean inputs. Inspired by this, we propose MASA, a method that utilizes
individual unlearning on local models to identify malicious models in FL. To
improve the performance of MASA in challenging non-independent and identically
distributed (non-IID) settings, we design pre-unlearning model fusion that
integrates local models with knowledge learned from other datasets to mitigate
the divergence in their unlearning behaviors caused by the non-IID data
distributions of clients. Additionally, we propose a new anomaly detection
metric with minimal hyperparameters to filter out malicious models efficiently.
Extensive experiments on IID and non-IID datasets across six different attacks
validate the effectiveness of MASA. To the best of our knowledge, this is the
first work to leverage machine unlearning to identify malicious models in FL.
Code is available at \url{https://github.com/JiiahaoXU/MASA}.


http://arxiv.org/abs/2411.01084
Plentiful Jailbreaks with String Compositions. (2%)
Brian R. Y. Huang
  Large language models (LLMs) remain vulnerable to a slew of adversarial
attacks and jailbreaking methods. One common approach employed by white-hat
attackers, or \textit{red-teamers}, is to process model inputs and outputs
using string-level obfuscations, which can include leetspeak, rotary ciphers,
Base64, ASCII, and more. Our work extends these encoding-based attacks by
unifying them in a framework of invertible string transformations. With
invertibility, we can devise arbitrary \textit{string compositions}, defined as
sequences of transformations, that we can encode and decode end-to-end
programmatically. We devise a automated best-of-n attack that samples from a
combinatorially large number of string compositions. Our jailbreaks obtain
competitive attack success rates on several leading frontier models when
evaluated on HarmBench, highlighting that encoding-based attacks remain a
persistent vulnerability even in advanced LLMs.


http://arxiv.org/abs/2411.00465
Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptions. (2%)
Rui Yang; Jie Wang; Guoping Wu; Bin Li
  Real-world offline datasets are often subject to data corruptions (such as
noise or adversarial attacks) due to sensor failures or malicious attacks.
Despite advances in robust offline reinforcement learning (RL), existing
methods struggle to learn robust agents under high uncertainty caused by the
diverse corrupted data (i.e., corrupted states, actions, rewards, and
dynamics), leading to performance degradation in clean environments. To tackle
this problem, we propose a novel robust variational Bayesian inference for
offline RL (TRACER). It introduces Bayesian inference for the first time to
capture the uncertainty via offline data for robustness against all types of
data corruptions. Specifically, TRACER first models all corruptions as the
uncertainty in the action-value function. Then, to capture such uncertainty, it
uses all offline data as the observations to approximate the posterior
distribution of the action-value function under a Bayesian inference framework.
An appealing feature of TRACER is that it can distinguish corrupted data from
clean data using an entropy-based uncertainty measure, since corrupted data
often induces higher uncertainty and entropy. Based on the aforementioned
measure, TRACER can regulate the loss associated with corrupted data to reduce
its influence, thereby enhancing robustness and performance in clean
environments. Experiments demonstrate that TRACER significantly outperforms
several state-of-the-art approaches across both individual and simultaneous
data corruptions.


http://arxiv.org/abs/2411.00349
Examining Attacks on Consensus and Incentive Systems in Proof-of-Work Blockchains: A Systematic Literature Review. (1%)
Dinitha Wijewardhana; Sugandima Vidanagamachchi; Nalin Arachchilage
  Cryptocurrencies have gained popularity due to their transparency, security,
and accessibility compared to traditional financial systems, with Bitcoin,
introduced in 2009, leading the market. Bitcoin's security relies on blockchain
technology - a decentralized ledger consisting of a consensus and an incentive
mechanism. The consensus mechanism, Proof of Work (PoW), requires miners to
solve difficult cryptographic puzzles to add new blocks, while the incentive
mechanism rewards them with newly minted bitcoins. However, as Bitcoin's
acceptance grows, it faces increasing threats from attacks targeting these
mechanisms, such as selfish mining, double-spending, and block withholding.
These attacks compromise security, efficiency, and reward distribution. Recent
research shows that these attacks can be combined with each other or with
either malicious strategies, such as network-layer attacks, or non-malicious
strategies, like honest mining. These combinations lead to more sophisticated
attacks, increasing the attacker's success rates and profitability. Therefore,
understanding and evaluating these attacks is essential for developing
effective countermeasures and ensuring long-term security. This paper begins by
examining individual attacks executed in isolation and their profitability. It
then explores how combining these attacks with each other or with other
malicious and non-malicious strategies can enhance their overall effectiveness
and profitability. The analysis further explores how the deployment of attacks
such as selfish mining and block withholding by multiple competing mining pools
against each other impacts their economic returns. Lastly, a set of design
guidelines is provided, outlining areas future work should focus on to prevent
or mitigate the identified threats.


http://arxiv.org/abs/2411.00715
B-cosification: Transforming Deep Neural Networks to be Inherently Interpretable. (1%)
Shreyash Arya; Sukrut Rao; Moritz Böhle; Bernt Schiele
  B-cos Networks have been shown to be effective for obtaining highly human
interpretable explanations of model decisions by architecturally enforcing
stronger alignment between inputs and weight. B-cos variants of convolutional
networks (CNNs) and vision transformers (ViTs), which primarily replace linear
layers with B-cos transformations, perform competitively to their respective
standard variants while also yielding explanations that are faithful by design.
However, it has so far been necessary to train these models from scratch, which
is increasingly infeasible in the era of large, pre-trained foundation models.
In this work, inspired by the architectural similarities in standard DNNs and
B-cos networks, we propose 'B-cosification', a novel approach to transform
existing pre-trained models to become inherently interpretable. We perform a
thorough study of design choices to perform this conversion, both for
convolutional neural networks and vision transformers. We find that
B-cosification can yield models that are on par with B-cos models trained from
scratch in terms of interpretability, while often outperforming them in terms
of classification performance at a fraction of the training cost. Subsequently,
we apply B-cosification to a pretrained CLIP model, and show that, even with
limited data and compute cost, we obtain a B-cosified version that is highly
interpretable and competitive on zero shot performance across a variety of
datasets. We release our code and pre-trained model weights at
https://github.com/shrebox/B-cosification.


http://arxiv.org/abs/2411.00403
Towards Building Secure UAV Navigation with FHE-aware Knowledge Distillation. (1%)
Arjun Ramesh Kaushik; Charanjit Jutla; Nalini Ratha
  In safeguarding mission-critical systems, such as Unmanned Aerial Vehicles
(UAVs), preserving the privacy of path trajectories during navigation is
paramount. While the combination of Reinforcement Learning (RL) and Fully
Homomorphic Encryption (FHE) holds promise, the computational overhead of FHE
presents a significant challenge. This paper proposes an innovative approach
that leverages Knowledge Distillation to enhance the practicality of secure UAV
navigation. By integrating RL and FHE, our framework addresses vulnerabilities
to adversarial attacks while enabling real-time processing of encrypted UAV
camera feeds, ensuring data security. To mitigate FHE's latency, Knowledge
Distillation is employed to compress the network, resulting in an impressive
18x speedup without compromising performance, as evidenced by an R-squared
score of 0.9499 compared to the original model's score of 0.9631. Our
methodology underscores the feasibility of processing encrypted data for UAV
navigation tasks, emphasizing security alongside performance efficiency and
timely processing. These findings pave the way for deploying autonomous UAVs in
sensitive environments, bolstering their resilience against potential security
threats.


http://arxiv.org/abs/2410.23870
Noise as a Double-Edged Sword: Reinforcement Learning Exploits Randomized Defenses in Neural Networks. (99%)
Steve Bakos; Pooria Madani; Heidar Davoudi
  This study investigates a counterintuitive phenomenon in adversarial machine
learning: the potential for noise-based defenses to inadvertently aid evasion
attacks in certain scenarios. While randomness is often employed as a defensive
strategy against adversarial examples, our research reveals that this approach
can sometimes backfire, particularly when facing adaptive attackers using
reinforcement learning (RL). Our findings show that in specific cases,
especially with visually noisy classes, the introduction of noise in the
classifier's confidence values can be exploited by the RL attacker, leading to
a significant increase in evasion success rates. In some instances, the
noise-based defense scenario outperformed other strategies by up to 20\% on a
subset of classes. However, this effect was not consistent across all
classifiers tested, highlighting the complexity of the interaction between
noise-based defenses and different models. These results suggest that in some
cases, noise-based defenses can inadvertently create an adversarial training
loop beneficial to the RL attacker. Our study emphasizes the need for a more
nuanced approach to defensive strategies in adversarial machine learning,
particularly in safety-critical applications. It challenges the assumption that
randomness universally enhances defense against evasion attacks and highlights
the importance of considering adaptive, RL-based attackers when designing
robust defense mechanisms.


http://arxiv.org/abs/2411.00222
Protecting Feed-Forward Networks from Adversarial Attacks Using Predictive Coding. (99%)
Ehsan Ganjidoost; Jeff Orchard
  An adversarial example is a modified input image designed to cause a Machine
Learning (ML) model to make a mistake; these perturbations are often invisible
or subtle to human observers and highlight vulnerabilities in a model's ability
to generalize from its training data. Several adversarial attacks can create
such examples, each with a different perspective, effectiveness, and
perceptibility of changes. Conversely, defending against such adversarial
attacks improves the robustness of ML models in image processing and other
domains of deep learning. Most defence mechanisms require either a level of
model awareness, changes to the model, or access to a comprehensive set of
adversarial examples during training, which is impractical. Another option is
to use an auxiliary model in a preprocessing manner without changing the
primary model. This study presents a practical and effective solution -- using
predictive coding networks (PCnets) as an auxiliary step for adversarial
defence. By seamlessly integrating PCnets into feed-forward networks as a
preprocessing step, we substantially bolster resilience to adversarial
perturbations. Our experiments on MNIST and CIFAR10 demonstrate the remarkable
effectiveness of PCnets in mitigating adversarial examples with about 82% and
65% improvements in robustness, respectively. The PCnet, trained on a small
subset of the dataset, leverages its generative nature to effectively counter
adversarial efforts, reverting perturbed images closer to their original forms.
This innovative approach holds promise for enhancing the security and
reliability of neural network classifiers in the face of the escalating threat
of adversarial attacks.


http://arxiv.org/abs/2410.23677
Wide Two-Layer Networks can Learn from Adversarial Perturbations. (98%)
Soichiro Kumano; Hiroshi Kera; Toshihiko Yamasaki
  Adversarial examples have raised several open questions, such as why they can
deceive classifiers and transfer between different models. A prevailing
hypothesis to explain these phenomena suggests that adversarial perturbations
appear as random noise but contain class-specific features. This hypothesis is
supported by the success of perturbation learning, where classifiers trained
solely on adversarial examples and the corresponding incorrect labels
generalize well to correctly labeled test data. Although this hypothesis and
perturbation learning are effective in explaining intriguing properties of
adversarial examples, their solid theoretical foundation is limited. In this
study, we theoretically explain the counterintuitive success of perturbation
learning. We assume wide two-layer networks and the results hold for any data
distribution. We prove that adversarial perturbations contain sufficient
class-specific features for networks to generalize from them. Moreover, the
predictions of classifiers trained on mislabeled adversarial examples coincide
with those of classifiers trained on correctly labeled clean samples. The code
is available at https://github.com/s-kumano/perturbation-learning.


http://arxiv.org/abs/2410.24006
DiffPAD: Denoising Diffusion-based Adversarial Patch Decontamination. (93%)
Jia Fu; Xiao Zhang; Sepideh Pashami; Fatemeh Rahimian; Anders Holst
  In the ever-evolving adversarial machine learning landscape, developing
effective defenses against patch attacks has become a critical challenge,
necessitating reliable solutions to safeguard real-world AI systems. Although
diffusion models have shown remarkable capacity in image synthesis and have
been recently utilized to counter $\ell_p$-norm bounded attacks, their
potential in mitigating localized patch attacks remains largely underexplored.
In this work, we propose DiffPAD, a novel framework that harnesses the power of
diffusion models for adversarial patch decontamination. DiffPAD first performs
super-resolution restoration on downsampled input images, then adopts
binarization, dynamic thresholding scheme and sliding window for effective
localization of adversarial patches. Such a design is inspired by the
theoretically derived correlation between patch size and diffusion restoration
error that is generalized across diverse patch attack scenarios. Finally,
DiffPAD applies inpainting techniques to the original input images with the
estimated patch region being masked. By integrating closed-form solutions for
super-resolution restoration and image inpainting into the conditional reverse
sampling process of a pre-trained diffusion model, DiffPAD obviates the need
for text guidance or fine-tuning. Through comprehensive experiments, we
demonstrate that DiffPAD not only achieves state-of-the-art adversarial
robustness against patch attacks but also excels in recovering naturalistic
images without patch remnants. The source code is available at
https://github.com/JasonFu1998/DiffPAD.


http://arxiv.org/abs/2411.00121
I Can Hear You: Selective Robust Training for Deepfake Audio Detection. (86%)
Zirui Zhang; Wei Hao; Aroon Sankoh; William Lin; Emanuel Mendiola-Ortiz; Junfeng Yang; Chengzhi Mao
  Recent advances in AI-generated voices have intensified the challenge of
detecting deepfake audio, posing risks for scams and the spread of
disinformation. To tackle this issue, we establish the largest public voice
dataset to date, named DeepFakeVox-HQ, comprising 1.3 million samples,
including 270,000 high-quality deepfake samples from 14 diverse sources.
Despite previously reported high accuracy, existing deepfake voice detectors
struggle with our diversely collected dataset, and their detection success
rates drop even further under realistic corruptions and adversarial attacks. We
conduct a holistic investigation into factors that enhance model robustness and
show that incorporating a diversified set of voice augmentations is beneficial.
Moreover, we find that the best detection models often rely on high-frequency
features, which are imperceptible to humans and can be easily manipulated by an
attacker. To address this, we propose the F-SAT: Frequency-Selective
Adversarial Training method focusing on high-frequency components. Empirical
results demonstrate that using our training dataset boosts baseline model
performance (without robust training) by 33%, and our robust training further
improves accuracy by 7.7% on clean samples and by 29.3% on corrupted and
attacked samples, over the state-of-the-art RawNet3 model.


http://arxiv.org/abs/2410.23678
Pseudo-Conversation Injection for LLM Goal Hijacking. (75%)
Zheng Chen; Buhui Yao
  Goal hijacking is a type of adversarial attack on Large Language Models
(LLMs) where the objective is to manipulate the model into producing a
specific, predetermined output, regardless of the user's original input. In
goal hijacking, an attacker typically appends a carefully crafted malicious
suffix to the user's prompt, which coerces the model into ignoring the user's
original input and generating the target response. In this paper, we introduce
a novel goal hijacking attack method called Pseudo-Conversation Injection,
which leverages the weaknesses of LLMs in role identification within
conversation contexts. Specifically, we construct the suffix by fabricating
responses from the LLM to the user's initial prompt, followed by a prompt for a
malicious new task. This leads the model to perceive the initial prompt and
fabricated response as a completed conversation, thereby executing the new,
falsified prompt. Following this approach, we propose three Pseudo-Conversation
construction strategies: Targeted Pseudo-Conversation, Universal
Pseudo-Conversation, and Robust Pseudo-Conversation. These strategies are
designed to achieve effective goal hijacking across various scenarios. Our
experiments, conducted on two mainstream LLM platforms including ChatGPT and
Qwen, demonstrate that our proposed method significantly outperforms existing
approaches in terms of attack effectiveness.


http://arxiv.org/abs/2410.24214
ARQ: A Mixed-Precision Quantization Framework for Accurate and Certifiably Robust DNNs. (41%)
Yuchen Yang; Shubham Ugare; Yifan Zhao; Gagandeep Singh; Sasa Misailovic
  Mixed precision quantization has become an important technique for enabling
the execution of deep neural networks (DNNs) on limited resource computing
platforms. Traditional quantization methods have primarily concentrated on
maintaining neural network accuracy, either ignoring the impact of quantization
on the robustness of the network, or using only empirical techniques for
improving robustness. In contrast, techniques for robustness certification,
which can provide strong guarantees about the robustness of DNNs have not been
used during quantization due to their high computation cost.
  This paper introduces ARQ, an innovative mixed-precision quantization method
that not only preserves the clean accuracy of the smoothed classifiers but also
maintains their certified robustness. ARQ uses reinforcement learning to find
accurate and robust DNN quantization, while efficiently leveraging randomized
smoothing, a popular class of statistical DNN verification algorithms, to guide
the search process.
  We compare ARQ with multiple state-of-the-art quantization techniques on
several DNN architectures commonly used in quantization studies: ResNet-20 on
CIFAR-10, ResNet-50 on ImageNet, and MobileNetV2 on ImageNet. We demonstrate
that ARQ consistently performs better than these baselines across all the
benchmarks and the input perturbation levels. In many cases, the performance of
ARQ quantized networks can reach that of the original DNN with floating-point
weights, but with only 1.5% instructions.


http://arxiv.org/abs/2411.00192
Optical Lens Attack on Monocular Depth Estimation for Autonomous Driving. (5%)
Ce Michigan State University Zhou; Qiben Michigan State University Yan; Daniel Michigan State University Kent; Guangjing University of South Florida Wang; Weikang Michigan State University Ding; Ziqi Peking University Zhang; Hayder Michigan State University Radha
  Monocular Depth Estimation (MDE) is a pivotal component of vision-based
Autonomous Driving (AD) systems, enabling vehicles to estimate the depth of
surrounding objects using a single camera image. This estimation guides
essential driving decisions, such as braking before an obstacle or changing
lanes to avoid collisions. In this paper, we explore vulnerabilities of MDE
algorithms in AD systems, presenting LensAttack, a novel physical attack that
strategically places optical lenses on the camera of an autonomous vehicle to
manipulate the perceived object depths. LensAttack encompasses two attack
formats: concave lens attack and convex lens attack, each utilizing different
optical lenses to induce false depth perception. We first develop a
mathematical model that outlines the parameters of the attack, followed by
simulations and real-world evaluations to assess its efficacy on
state-of-the-art MDE models. Additionally, we adopt an attack optimization
method to further enhance the attack success rate by optimizing the attack
focal length. To better evaluate the implications of LensAttack on AD, we
conduct comprehensive end-to-end system simulations using the CARLA platform.
The results reveal that LensAttack can significantly disrupt the depth
estimation processes in AD systems, posing a serious threat to their
reliability and safety. Finally, we discuss some potential defense methods to
mitigate the effects of the proposed attack.


http://arxiv.org/abs/2410.23687
Adversarial Attacks of Vision Tasks in the Past 10 Years: A Survey. (2%)
Chiyu Zhang; Xiaogang Xu; Jiafei Wu; Zhe Liu; Lu Zhou
  Adversarial attacks, which manipulate input data to undermine model
availability and integrity, pose significant security threats during machine
learning inference. With the advent of Large Vision-Language Models (LVLMs),
new attack vectors, such as cognitive bias, prompt injection, and jailbreak
techniques, have emerged. Understanding these attacks is crucial for developing
more robust systems and demystifying the inner workings of neural networks.
However, existing reviews often focus on attack classifications and lack
comprehensive, in-depth analysis. The research community currently needs: 1)
unified insights into adversariality, transferability, and generalization; 2)
detailed evaluations of existing methods; 3) motivation-driven attack
categorizations; and 4) an integrated perspective on both traditional and LVLM
attacks. This article addresses these gaps by offering a thorough summary of
traditional and LVLM adversarial attacks, emphasizing their connections and
distinctions, and providing actionable insights for future research.


http://arxiv.org/abs/2410.23142
FAIR-TAT: Improving Model Fairness Using Targeted Adversarial Training. (99%)
Tejaswini Medi; Steffen Jung; Margret Keuper
  Deep neural networks are susceptible to adversarial attacks and common
corruptions, which undermine their robustness. In order to enhance model
resilience against such challenges, Adversarial Training (AT) has emerged as a
prominent solution. Nevertheless, adversarial robustness is often attained at
the expense of model fairness during AT, i.e., disparity in class-wise
robustness of the model. While distinctive classes become more robust towards
such adversaries, hard to detect classes suffer. Recently, research has focused
on improving model fairness specifically for perturbed images, overlooking the
accuracy of the most likely non-perturbed data. Additionally, despite their
robustness against the adversaries encountered during model training,
state-of-the-art adversarial trained models have difficulty maintaining
robustness and fairness when confronted with diverse adversarial threats or
common corruptions. In this work, we address the above concerns by introducing
a novel approach called Fair Targeted Adversarial Training (FAIR-TAT). We show
that using targeted adversarial attacks for adversarial training (instead of
untargeted attacks) can allow for more favorable trade-offs with respect to
adversarial fairness. Empirical results validate the efficacy of our approach.


http://arxiv.org/abs/2410.23483
Keep on Swimming: Real Attackers Only Need Partial Knowledge of a Multi-Model System. (99%)
Julian Collado; Kevin Stangl
  Recent approaches in machine learning often solve a task using a composition
of multiple models or agentic architectures. When targeting a composed system
with adversarial attacks, it might not be computationally or informationally
feasible to train an end-to-end proxy model or a proxy model for every
component of the system. We introduce a method to craft an adversarial attack
against the overall multi-model system when we only have a proxy model for the
final black-box model, and when the transformation applied by the initial
models can make the adversarial perturbations ineffective. Current methods
handle this by applying many copies of the first model/transformation to an
input and then re-use a standard adversarial attack by averaging gradients, or
learning a proxy model for both stages. To our knowledge, this is the first
attack specifically designed for this threat model and our method has a
substantially higher attack success rate (80% vs 25%) and contains 9.4% smaller
perturbations (MSE) compared to prior state-of-the-art methods. Our experiments
focus on a supervised image pipeline, but we are confident the attack will
generalize to other multi-model settings [e.g. a mix of open/closed source
foundation models], or agentic systems


http://arxiv.org/abs/2410.23091
CausalDiff: Causality-Inspired Disentanglement via Diffusion Model for Adversarial Defense. (99%)
Mingkun Zhang; Keping Bi; Wei Chen; Quanrun Chen; Jiafeng Guo; Xueqi Cheng
  Despite ongoing efforts to defend neural classifiers from adversarial
attacks, they remain vulnerable, especially to unseen attacks. In contrast,
humans are difficult to be cheated by subtle manipulations, since we make
judgments only based on essential factors. Inspired by this observation, we
attempt to model label generation with essential label-causative factors and
incorporate label-non-causative factors to assist data generation. For an
adversarial example, we aim to discriminate the perturbations as non-causative
factors and make predictions only based on the label-causative factors.
Concretely, we propose a casual diffusion model (CausalDiff) that adapts
diffusion models for conditional data generation and disentangles the two types
of casual factors by learning towards a novel casual information bottleneck
objective. Empirically, CausalDiff has significantly outperformed
state-of-the-art defense methods on various unseen attacks, achieving an
average robustness of 86.39% (+4.01%) on CIFAR-10, 56.25% (+3.13%) on
CIFAR-100, and 82.62% (+4.93%) on GTSRB (German Traffic Sign Recognition
Benchmark). The code is available at
\href{https://github.com/CAS-AISafetyBasicResearchGroup/CausalDiff}{https://github.com/CAS-AISafetyBasicResearchGroup/CausalDiff}


http://arxiv.org/abs/2410.22725
One Prompt to Verify Your Models: Black-Box Text-to-Image Models Verification via Non-Transferable Adversarial Attacks. (98%)
Ji Guo; Wenbo Jiang; Rui Zhang; Guoming Lu; Hongwei Li
  Recently, the success of Text-to-Image (T2I) models has led to the rise of
numerous third-party platforms, which claim to provide cheaper API services and
more flexibility in model options. However, this also raises a new security
concern: Are these third-party services truly offering the models they claim?
To address this problem, we propose the first T2I model verification method
named Text-to-Image Model Verification via Non-Transferable Adversarial Attacks
(TVN). The non-transferability of adversarial examples means that these
examples are only effective on a target model and ineffective on other models,
thereby allowing for the verification of the target model. TVN utilizes the
Non-dominated Sorting Genetic Algorithm II (NSGA-II) to optimize the cosine
similarity of a prompt's text encoding, generating non-transferable adversarial
prompts. By calculating the CLIP-text scores between the non-transferable
adversarial prompts without perturbations and the images, we can verify if the
model matches the claimed target model, based on a 3-sigma threshold. The
experiments showed that TVN performed well in both closed-set and open-set
scenarios, achieving a verification accuracy of over 90\%. Moreover, the
adversarial prompts generated by TVN significantly reduced the CLIP-text scores
of the target model, while having little effect on other models.


http://arxiv.org/abs/2410.22888
Effective and Efficient Adversarial Detection for Vision-Language Models via A Single Vector. (87%)
Youcheng Huang; Fengbin Zhu; Jingkun Tang; Pan Zhou; Wenqiang Lei; Jiancheng Lv; Tat-Seng Chua
  Visual Language Models (VLMs) are vulnerable to adversarial attacks,
especially those from adversarial images, which is however under-explored in
literature. To facilitate research on this critical safety problem, we first
construct a new laRge-scale Adervsarial images dataset with Diverse hArmful
Responses (RADAR), given that existing datasets are either small-scale or only
contain limited types of harmful responses. With the new RADAR dataset, we
further develop a novel and effective iN-time Embedding-based AdveRSarial Image
DEtection (NEARSIDE) method, which exploits a single vector that distilled from
the hidden states of VLMs, which we call the attacking direction, to achieve
the detection of adversarial images against benign ones in the input. Extensive
experiments with two victim VLMs, LLaVA and MiniGPT-4, well demonstrate the
effectiveness, efficiency, and cross-model transferrability of our proposed
method. Our code is available at https://github.com/mob-scu/RADAR-NEARSIDE


http://arxiv.org/abs/2410.22832
HijackRAG: Hijacking Attacks against Retrieval-Augmented Large Language Models. (82%)
Yucheng Zhang; Qinfeng Li; Tianyu Du; Xuhong Zhang; Xinkui Zhao; Zhengwen Feng; Jianwei Yin
  Retrieval-Augmented Generation (RAG) systems enhance large language models
(LLMs) by integrating external knowledge, making them adaptable and
cost-effective for various applications. However, the growing reliance on these
systems also introduces potential security risks. In this work, we reveal a
novel vulnerability, the retrieval prompt hijack attack (HijackRAG), which
enables attackers to manipulate the retrieval mechanisms of RAG systems by
injecting malicious texts into the knowledge database. When the RAG system
encounters target questions, it generates the attacker's pre-determined answers
instead of the correct ones, undermining the integrity and trustworthiness of
the system. We formalize HijackRAG as an optimization problem and propose both
black-box and white-box attack strategies tailored to different levels of the
attacker's knowledge. Extensive experiments on multiple benchmark datasets show
that HijackRAG consistently achieves high attack success rates, outperforming
existing baseline attacks. Furthermore, we demonstrate that the attack is
transferable across different retriever models, underscoring the widespread
risk it poses to RAG systems. Lastly, our exploration of various defense
mechanisms reveals that they are insufficient to counter HijackRAG, emphasizing
the urgent need for more robust security measures to protect RAG systems in
real-world deployments.


http://arxiv.org/abs/2410.22844
Understanding and Improving Adversarial Collaborative Filtering for Robust Recommendation. (67%)
Kaike Zhang; Qi Cao; Yunfan Wu; Fei Sun; Huawei Shen; Xueqi Cheng
  Adversarial Collaborative Filtering (ACF), which typically applies
adversarial perturbations at user and item embeddings through adversarial
training, is widely recognized as an effective strategy for enhancing the
robustness of Collaborative Filtering (CF) recommender systems against
poisoning attacks. Besides, numerous studies have empirically shown that ACF
can also improve recommendation performance compared to traditional CF. Despite
these empirical successes, the theoretical understanding of ACF's effectiveness
in terms of both performance and robustness remains unclear. To bridge this
gap, in this paper, we first theoretically show that ACF can achieve a lower
recommendation error compared to traditional CF with the same training epochs
in both clean and poisoned data contexts. Furthermore, by establishing bounds
for reductions in recommendation error during ACF's optimization process, we
find that applying personalized magnitudes of perturbation for different users
based on their embedding scales can further improve ACF's effectiveness.
Building on these theoretical understandings, we propose Personalized Magnitude
Adversarial Collaborative Filtering (PamaCF). Extensive experiments demonstrate
that PamaCF effectively defends against various types of poisoning attacks
while significantly enhancing recommendation performance.


http://arxiv.org/abs/2410.23118
Teaching a Language Model to Distinguish Between Similar Details using a Small Adversarial Training Set. (64%)
Chris Achard
  Language models can achieve high accuracy on natural language tasks such as
NLI, but performance suffers on manually created adversarial examples. We
investigate the performance of a language model trained on the Stanford Natural
Language Inference (SNLI) corpus on a manually created adversarial test set. We
then improve the model's performance by fine tuning the model on a small,
manually created adversarial training set, designed to help the language model
to learn to differentiate between similar words and phrases in the data. We
show an increase in accuracy on the adversarial test set (+ 13%) while still
maintaining good performance on the original NLI task. We also show an increase
in accuracy from 91.2% to 92.9% on the most similar contradictions in the SNLI
test set (as judged by cosine similarity).


http://arxiv.org/abs/2410.22678
Backdoor Attack Against Vision Transformers via Attention Gradient-Based Image Erosion. (62%)
Ji Guo; Hongwei Li; Wenbo Jiang; Guoming Lu
  Vision Transformers (ViTs) have outperformed traditional Convolutional Neural
Networks (CNN) across various computer vision tasks. However, akin to CNN, ViTs
are vulnerable to backdoor attacks, where the adversary embeds the backdoor
into the victim model, causing it to make wrong predictions about testing
samples containing a specific trigger. Existing backdoor attacks against ViTs
have the limitation of failing to strike an optimal balance between attack
stealthiness and attack effectiveness.
  In this work, we propose an Attention Gradient-based Erosion Backdoor (AGEB)
targeted at ViTs. Considering the attention mechanism of ViTs, AGEB selectively
erodes pixels in areas of maximal attention gradient, embedding a covert
backdoor trigger. Unlike previous backdoor attacks against ViTs, AGEB achieves
an optimal balance between attack stealthiness and attack effectiveness,
ensuring the trigger remains invisible to human detection while preserving the
model's accuracy on clean samples. Extensive experimental evaluations across
various ViT architectures and datasets confirm the effectiveness of AGEB,
achieving a remarkable Attack Success Rate (ASR) without diminishing Clean Data
Accuracy (CDA). Furthermore, the stealthiness of AGEB is rigorously validated,
demonstrating minimal visual discrepancies between the clean and the triggered
images.


http://arxiv.org/abs/2410.22705
Geometry Cloak: Preventing TGS-based 3D Reconstruction from Copyrighted Images. (2%)
Qi Song; Ziyuan Luo; Ka Chun Cheung; Simon See; Renjie Wan
  Single-view 3D reconstruction methods like Triplane Gaussian Splatting (TGS)
have enabled high-quality 3D model generation from just a single image input
within seconds. However, this capability raises concerns about potential
misuse, where malicious users could exploit TGS to create unauthorized 3D
models from copyrighted images. To prevent such infringement, we propose a
novel image protection approach that embeds invisible geometry perturbations,
termed "geometry cloaks", into images before supplying them to TGS. These
carefully crafted perturbations encode a customized message that is revealed
when TGS attempts 3D reconstructions of the cloaked image. Unlike conventional
adversarial attacks that simply degrade output quality, our method forces TGS
to fail the 3D reconstruction in a specific way - by generating an identifiable
customized pattern that acts as a watermark. This watermark allows copyright
holders to assert ownership over any attempted 3D reconstructions made from
their protected images. Extensive experiments have verified the effectiveness
of our geometry cloak. Our project is available at
https://qsong2001.github.io/geometry_cloak.


http://arxiv.org/abs/2410.22680
Byzantine-Robust Federated Learning: An Overview With Focus on Developing Sybil-based Attacks to Backdoor Augmented Secure Aggregation Protocols. (1%)
Atharv Deshmukh
  Federated Learning (FL) paradigms enable large numbers of clients to
collaboratively train Machine Learning models on private data. However, due to
their multi-party nature, traditional FL schemes are left vulnerable to
Byzantine attacks that attempt to hurt model performance by injecting malicious
backdoors. A wide variety of prevention methods have been proposed to protect
frameworks from such attacks. This paper provides a exhaustive and updated
taxonomy of existing methods and frameworks, before zooming in and conducting
an in-depth analysis of the strengths and weaknesses of the Robustness of
Federated Learning (RoFL) protocol. From there, we propose two novel
Sybil-based attacks that take advantage of vulnerabilities in RoFL. Finally, we
conclude with comprehensive proposals for future testing, describe and detail
implementation of the proposed attacks, and offer direction for improvements in
the RoFL protocol as well as Byzantine-robust frameworks as a whole.


http://arxiv.org/abs/2410.23182
ProTransformer: Robustify Transformers via Plug-and-Play Paradigm. (1%)
Zhichao Hou; Weizhi Gao; Yuchen Shen; Feiyi Wang; Xiaorui Liu
  Transformer-based architectures have dominated various areas of machine
learning in recent years. In this paper, we introduce a novel robust attention
mechanism designed to enhance the resilience of transformer-based
architectures. Crucially, this technique can be integrated into existing
transformers as a plug-and-play layer, improving their robustness without the
need for additional training or fine-tuning. Through comprehensive experiments
and ablation studies, we demonstrate that our ProTransformer significantly
enhances the robustness of transformer models across a variety of prediction
tasks, attack mechanisms, backbone architectures, and data domains. Notably,
without further fine-tuning, the ProTransformer consistently improves the
performance of vanilla transformers by 19.5%, 28.3%, 16.1%, and 11.4% for BERT,
ALBERT, DistilBERT, and RoBERTa, respectively, under the classical TextFooler
attack. Furthermore, ProTransformer shows promising resilience in large
language models (LLMs) against prompting-based attacks, improving the
performance of T5 and LLaMA by 24.8% and 17.8%, respectively, and enhancing
Vicuna by an average of 10.4% against the Jailbreaking attack. Beyond the
language domain, ProTransformer also demonstrates outstanding robustness in
both vision and graph domains.


http://arxiv.org/abs/2410.23232
Attribute-to-Delete: Machine Unlearning via Datamodel Matching. (1%)
Kristian Georgiev; Roy Rinberg; Sung Min Park; Shivam Garg; Andrew Ilyas; Aleksander Madry; Seth Neel
  Machine unlearning -- efficiently removing the effect of a small "forget set"
of training data on a pre-trained machine learning model -- has recently
attracted significant research interest. Despite this interest, however, recent
work shows that existing machine unlearning techniques do not hold up to
thorough evaluation in non-convex settings. In this work, we introduce a new
machine unlearning technique that exhibits strong empirical performance even in
such challenging settings. Our starting point is the perspective that the goal
of unlearning is to produce a model whose outputs are statistically
indistinguishable from those of a model re-trained on all but the forget set.
This perspective naturally suggests a reduction from the unlearning problem to
that of data attribution, where the goal is to predict the effect of changing
the training set on a model's outputs. Thus motivated, we propose the following
meta-algorithm, which we call Datamodel Matching (DMM): given a trained model,
we (a) use data attribution to predict the output of the model if it were
re-trained on all but the forget set points; then (b) fine-tune the pre-trained
model to match these predicted outputs. In a simple convex setting, we show how
this approach provably outperforms a variety of iterative unlearning
algorithms. Empirically, we use a combination of existing evaluations and a new
metric based on the KL-divergence to show that even in non-convex settings, DMM
achieves strong unlearning performance relative to existing algorithms. An
added benefit of DMM is that it is a meta-algorithm, in the sense that future
advances in data attribution translate directly into better unlearning
algorithms, pointing to a clear direction for future progress in unlearning.


http://arxiv.org/abs/2410.22884
Stealing User Prompts from Mixture of Experts. (1%)
Itay Yona; Ilia Shumailov; Jamie Hayes; Nicholas Carlini
  Mixture-of-Experts (MoE) models improve the efficiency and scalability of
dense language models by routing each token to a small number of experts in
each layer. In this paper, we show how an adversary that can arrange for their
queries to appear in the same batch of examples as a victim's queries can
exploit Expert-Choice-Routing to fully disclose a victim's prompt. We
successfully demonstrate the effectiveness of this attack on a two-layer
Mixtral model, exploiting the tie-handling behavior of the torch.topk CUDA
implementation. Our results show that we can extract the entire prompt using
$O({VM}^2)$ queries (with vocabulary size $V$ and prompt length $M$) or 100
queries on average per token in the setting we consider. This is the first
attack to exploit architectural flaws for the purpose of extracting user
prompts, introducing a new class of LLM vulnerabilities.


http://arxiv.org/abs/2410.22770
InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Models. (1%)
Hao Li; Xiaogeng Liu; Chaowei Xiao
  Prompt injection attacks pose a critical threat to large language models
(LLMs), enabling goal hijacking and data leakage. Prompt guard models, though
effective in defense, suffer from over-defense -- falsely flagging benign
inputs as malicious due to trigger word bias. To address this issue, we
introduce NotInject, an evaluation dataset that systematically measures
over-defense across various prompt guard models. NotInject contains 339 benign
samples enriched with trigger words common in prompt injection attacks,
enabling fine-grained evaluation. Our results show that state-of-the-art models
suffer from over-defense issues, with accuracy dropping close to random
guessing levels (60%). To mitigate this, we propose InjecGuard, a novel prompt
guard model that incorporates a new training strategy, Mitigating Over-defense
for Free (MOF), which significantly reduces the bias on trigger words.
InjecGuard demonstrates state-of-the-art performance on diverse benchmarks
including NotInject, surpassing the existing best model by 30.8%, offering a
robust and open-source solution for detecting prompt injection attacks. The
code and datasets are released at https://github.com/SaFoLab-WISC/InjecGuard.


http://arxiv.org/abs/2410.21952
On the Robustness of Adversarial Training Against Uncertainty Attacks. (99%)
Emanuele Ledda; Giovanni Scodeller; Daniele Angioni; Giorgio Piras; Antonio Emanuele Cinà; Giorgio Fumera; Battista Biggio; Fabio Roli
  In learning problems, the noise inherent to the task at hand hinders the
possibility to infer without a certain degree of uncertainty. Quantifying this
uncertainty, regardless of its wide use, assumes high relevance for
security-sensitive applications. Within these scenarios, it becomes fundamental
to guarantee good (i.e., trustworthy) uncertainty measures, which downstream
modules can securely employ to drive the final decision-making process.
However, an attacker may be interested in forcing the system to produce either
(i) highly uncertain outputs jeopardizing the system's availability or (ii) low
uncertainty estimates, making the system accept uncertain samples that would
instead require a careful inspection (e.g., human intervention). Therefore, it
becomes fundamental to understand how to obtain robust uncertainty estimates
against these kinds of attacks. In this work, we reveal both empirically and
theoretically that defending against adversarial examples, i.e., carefully
perturbed samples that cause misclassification, additionally guarantees a more
secure, trustworthy uncertainty estimate under common attack scenarios without
the need for an ad-hoc defense strategy. To support our claims, we evaluate
multiple adversarial-robust models from the publicly available benchmark
RobustBench on the CIFAR-10 and ImageNet datasets.


http://arxiv.org/abs/2411.00839
CausAdv: A Causal-based Framework for Detecting Adversarial Examples. (99%)
Hichem Debbi
  Deep learning has led to tremendous success in many real-world applications
of computer vision, thanks to sophisticated architectures such as Convolutional
neural networks (CNNs). However, CNNs have been shown to be vulnerable to
crafted adversarial perturbations in inputs. These inputs appear almost
indistinguishable from natural images, yet they are incorrectly classified by
CNN architectures. This vulnerability of adversarial examples has led
researchers to focus on enhancing the robustness of deep learning models in
general, and CNNs in particular, by creating defense and detection methods to
distinguish adversarials inputs from natural ones. In this paper, we address
the adversarial robustness of CNNs through causal reasoning.
  We propose CausAdv: a causal framework for detecting adversarial examples
based on counterfactual reasoning. CausAdv learns causal and non-causal
features of every input, and quantifies the counterfactual information (CI) of
every filter of the last convolutional layer. Then we perform statistical
analysis on the filters CI of every sample, whether clan or adversarials, to
demonstrate how adversarial examples indeed exhibit different CI distributions
compared to clean samples. Our results show that causal reasoning enhances the
process of adversarials detection without the need to train a separate
detector. In addition, we illustrate the efficiency of causal explanations as a
helpful detection technique through visualizing the causal features. The
results can be reproduced using the code available in the repository:
https://github.com/HichemDebbi/CausAdv.


http://arxiv.org/abs/2410.21802
Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models. (98%)
Lu Yu; Haiyang Zhang; Changsheng Xu
  Due to the impressive zero-shot capabilities, pre-trained vision-language
models (e.g. CLIP), have attracted widespread attention and adoption across
various domains. Nonetheless, CLIP has been observed to be susceptible to
adversarial examples. Through experimental analysis, we have observed a
phenomenon wherein adversarial perturbations induce shifts in text-guided
attention. Building upon this observation, we propose a simple yet effective
strategy: Text-Guided Attention for Zero-Shot Robustness (TGA-ZSR). This
framework incorporates two components: the Attention Refinement module and the
Attention-based Model Constraint module. Our goal is to maintain the
generalization of the CLIP model and enhance its adversarial robustness: The
Attention Refinement module aligns the text-guided attention obtained from the
target model via adversarial examples with the text-guided attention acquired
from the original model via clean examples. This alignment enhances the model's
robustness. Additionally, the Attention-based Model Constraint module acquires
text-guided attention from both the target and original models using clean
examples. Its objective is to maintain model performance on clean samples while
enhancing overall robustness. The experiments validate that our method yields a
9.58% enhancement in zero-shot robust accuracy over the current
state-of-the-art techniques across 16 datasets. Our code is available at
https://github.com/zhyblue424/TGA-ZSR.


http://arxiv.org/abs/2411.00827
IDEATOR: Jailbreaking Large Vision-Language Models Using Themselves. (83%)
Ruofan Wang; Bo Wang; Xiaosen Wang; Xingjun Ma; Yu-Gang Jiang
  As large Vision-Language Models (VLMs) grow in prominence, ensuring their
safe deployment has become critical. Recent studies have explored VLM
robustness against jailbreak attacks--techniques that exploit model
vulnerabilities to elicit harmful outputs. However, the limited availability of
diverse multi-modal data has led current approaches to rely heavily on
adversarial or manually crafted images derived from harmful text datasets,
which may lack effectiveness and diversity across different contexts. In this
paper, we propose a novel jailbreak method named IDEATOR, which autonomously
generates malicious image-text pairs for black-box jailbreak attacks. IDEATOR
is based on the insight that VLMs themselves could serve as powerful red team
models for generating multimodal jailbreak prompts. Specifically, IDEATOR uses
a VLM to create targeted jailbreak texts and pairs them with jailbreak images
generated by a state-of-the-art diffusion model. Our extensive experiments
demonstrate IDEATOR's high effectiveness and transferability. Notably, it
achieves a 94% success rate in jailbreaking MiniGPT-4 with an average of only
5.34 queries, and high success rates of 82%, 88%, and 75% when transferred to
LLaVA, InstructBLIP, and Meta's Chameleon, respectively. IDEATOR uncovers
specific vulnerabilities in VLMs under black-box conditions, underscoring the
need for improved safety mechanisms.


http://arxiv.org/abs/2410.22663
Automated Trustworthiness Oracle Generation for Machine Learning Text Classifiers. (82%)
Lam Nguyen Tung; Steven Cho; Xiaoning Du; Neelofar Neelofar; Valerio Terragni; Stefano Ruberto; Aldeida Aleti
  Machine learning (ML) for text classification has been widely used in various
domains, such as toxicity detection, chatbot consulting, and review analysis.
These applications can significantly impact ethics, economics, and human
behavior, raising serious concerns about trusting ML decisions. Several studies
indicate that traditional metrics, such as model confidence and accuracy, are
insufficient to build human trust in ML models. These models often learn
spurious correlations during training and predict based on them during
inference. In the real world, where such correlations are absent, their
performance can deteriorate significantly. To avoid this, a common practice is
to test whether predictions are reasonable. Along with this, a challenge known
as the trustworthiness oracle problem has been introduced. Due to the lack of
automated trustworthiness oracles, the assessment requires manual validation of
the decision process disclosed by explanation methods, which is time-consuming
and not scalable. We propose TOKI, the first automated trustworthiness oracle
generation method for text classifiers, which automatically checks whether the
prediction-contributing words are related to the predicted class using
explanation methods and word embeddings. To demonstrate its practical
usefulness, we introduce a novel adversarial attack method targeting
trustworthiness issues identified by TOKI. We compare TOKI with a naive
baseline based solely on model confidence using human-created ground truths of
6,000 predictions. We also compare TOKI-guided adversarial attack method with
A2T, a SOTA adversarial attack method. Results show that relying on prediction
uncertainty cannot distinguish between trustworthy and untrustworthy
predictions, TOKI achieves 142% higher accuracy than the naive baseline, and
TOKI-guided adversarial attack method is more effective with fewer
perturbations than A2T.


http://arxiv.org/abs/2411.00837
Longitudinal Mammogram Exam-based Breast Cancer Diagnosis Models: Vulnerability to Adversarial Attacks. (81%)
Zhengbo Zhou; Degan Hao; Dooman Arefan; Margarita Zuley; Jules Sumkin; Shandong Wu
  In breast cancer detection and diagnosis, the longitudinal analysis of
mammogram images is crucial. Contemporary models excel in detecting temporal
imaging feature changes, thus enhancing the learning process over sequential
imaging exams. Yet, the resilience of these longitudinal models against
adversarial attacks remains underexplored. In this study, we proposed a novel
attack method that capitalizes on the feature-level relationship between two
sequential mammogram exams of a longitudinal model, guided by both
cross-entropy loss and distance metric learning, to achieve significant attack
efficacy, as implemented using attack transferring in a black-box attacking
manner. We performed experiments on a cohort of 590 breast cancer patients
(each has two sequential mammogram exams) in a case-control setting. Results
showed that our proposed method surpassed several state-of-the-art adversarial
attacks in fooling the diagnosis models to give opposite outputs. Our method
remained effective even if the model was trained with the common defending
method of adversarial training.


http://arxiv.org/abs/2410.22143
AmpleGCG-Plus: A Strong Generative Model of Adversarial Suffixes to Jailbreak LLMs with Higher Success Rates in Fewer Attempts. (78%)
Vishal Kumar; Zeyi Liao; Jaylen Jones; Huan Sun
  Although large language models (LLMs) are typically aligned, they remain
vulnerable to jailbreaking through either carefully crafted prompts in natural
language or, interestingly, gibberish adversarial suffixes. However, gibberish
tokens have received relatively less attention despite their success in
attacking aligned LLMs. Recent work, AmpleGCG~\citep{liao2024amplegcg},
demonstrates that a generative model can quickly produce numerous customizable
gibberish adversarial suffixes for any harmful query, exposing a range of
alignment gaps in out-of-distribution (OOD) language spaces. To bring more
attention to this area, we introduce AmpleGCG-Plus, an enhanced version that
achieves better performance in fewer attempts. Through a series of exploratory
experiments, we identify several training strategies to improve the learning of
gibberish suffixes. Our results, verified under a strict evaluation setting,
show that it outperforms AmpleGCG on both open-weight and closed-source models,
achieving increases in attack success rate (ASR) of up to 17\% in the white-box
setting against Llama-2-7B-chat, and more than tripling ASR in the black-box
setting against GPT-4. Notably, AmpleGCG-Plus jailbreaks the newer GPT-4o
series of models at similar rates to GPT-4, and, uncovers vulnerabilities
against the recently proposed circuit breakers defense. We publicly release
AmpleGCG-Plus along with our collected training datasets.


http://arxiv.org/abs/2410.22284
Embedding-based classifiers can detect prompt injection attacks. (64%)
Md. Ahsan Ayub; Subhabrata Majumdar
  Large Language Models (LLMs) are seeing significant adoption in every type of
organization due to their exceptional generative capabilities. However, LLMs
are found to be vulnerable to various adversarial attacks, particularly prompt
injection attacks, which trick them into producing harmful or inappropriate
content. Adversaries execute such attacks by crafting malicious prompts to
deceive the LLMs. In this paper, we propose a novel approach based on
embedding-based Machine Learning (ML) classifiers to protect LLM-based
applications against this severe threat. We leverage three commonly used
embedding models to generate embeddings of malicious and benign prompts and
utilize ML classifiers to predict whether an input prompt is malicious. Out of
several traditional ML methods, we achieve the best performance with
classifiers built using Random Forest and XGBoost. Our classifiers outperform
state-of-the-art prompt injection classifiers available in open-source
implementations, which use encoder-only neural networks.


http://arxiv.org/abs/2410.21791
Enhancing Adversarial Attacks through Chain of Thought. (54%)
Jingbo Su
  Large language models (LLMs) have demonstrated impressive performance across
various domains but remain susceptible to safety concerns. Prior research
indicates that gradient-based adversarial attacks are particularly effective
against aligned LLMs and the chain of thought (CoT) prompting can elicit
desired answers through step-by-step reasoning. This paper proposes enhancing
the robustness of adversarial attacks on aligned LLMs by integrating CoT
prompts with the greedy coordinate gradient (GCG) technique. Using CoT triggers
instead of affirmative targets stimulates the reasoning abilities of backend
LLMs, thereby improving the transferability and universality of adversarial
attacks. We conducted an ablation study comparing our CoT-GCG approach with
Amazon Web Services auto-cot. Results revealed our approach outperformed both
the baseline GCG attack and CoT prompting. Additionally, we used Llama Guard to
evaluate potentially harmful interactions, providing a more objective risk
assessment of entire conversations compared to matching outputs to rejection
phrases. The code of this paper is available at
https://github.com/sujingbo0217/CS222W24-LLM-Attack.


http://arxiv.org/abs/2410.22425
Power side-channel leakage localization through adversarial training of deep neural networks. (11%)
Jimmy Gammell; Anand Raghunathan; Kaushik Roy
  Supervised deep learning has emerged as an effective tool for carrying out
power side-channel attacks on cryptographic implementations. While
increasingly-powerful deep learning-based attacks are regularly published,
comparatively-little work has gone into using deep learning to defend against
these attacks. In this work we propose a technique for identifying which
timesteps in a power trace are responsible for leaking a cryptographic key,
through an adversarial game between a deep learning-based side-channel attacker
which seeks to classify a sensitive variable from the power traces recorded
during encryption, and a trainable noise generator which seeks to thwart this
attack by introducing a minimal amount of noise into the power traces. We
demonstrate on synthetic datasets that our method can outperform existing
techniques in the presence of common countermeasures such as Boolean masking
and trace desynchronization. Results on real datasets are weak because the
technique is highly sensitive to hyperparameters and early-stop point, and we
lack a holdout dataset with ground truth knowledge of leaking points for model
selection. Nonetheless, we believe our work represents an important first step
towards deep side-channel leakage localization without relying on strong
assumptions about the implementation or the nature of its leakage. An
open-source PyTorch implementation of our experiments is provided.


http://arxiv.org/abs/2410.21736
Enhancing Safety and Robustness of Vision-Based Controllers via Reachability Analysis. (1%)
Kaustav Chakraborty; Aryaman Gupta; Somil Bansal
  Autonomous systems, such as self-driving cars and drones, have made
significant strides in recent years by leveraging visual inputs and machine
learning for decision-making and control. Despite their impressive performance,
these vision-based controllers can make erroneous predictions when faced with
novel or out-of-distribution inputs. Such errors can cascade into catastrophic
system failures and compromise system safety. In this work, we compute Neural
Reachable Tubes, which act as parameterized approximations of Backward
Reachable Tubes to stress-test the vision-based controllers and mine their
failure modes. The identified failures are then used to enhance the system
safety through both offline and online methods. The online approach involves
training a classifier as a run-time failure monitor to detect closed-loop,
system-level failures, subsequently triggering a fallback controller that
robustly handles these detected failures to preserve system safety. For the
offline approach, we improve the original controller via incremental training
using a carefully augmented failure dataset, resulting in a more robust
controller that is resistant to the known failure modes. In either approach,
the system is safeguarded against shortcomings that transcend the vision-based
controller and pertain to the closed-loop safety of the overall system. We
validate the proposed approaches on an autonomous aircraft taxiing task that
involves using a vision-based controller to guide the aircraft towards the
centerline of the runway. Our results show the efficacy of the proposed
algorithms in identifying and handling system-level failures, outperforming
methods that rely on controller prediction error or uncertainty quantification
for identifying system failures.


http://arxiv.org/abs/2411.00836
DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models. (1%)
Chengke Zou; Xingang Guo; Rui Yang; Junyu Zhang; Bin Hu; Huan Zhang
  The rapid advancements in Vision-Language Models (VLMs) have shown great
potential in tackling mathematical reasoning tasks that involve visual context.
Unlike humans who can reliably apply solution steps to similar problems with
minor modifications, we found that SOTA VLMs like GPT-4o can consistently fail
in these scenarios, revealing limitations in their mathematical reasoning
capabilities. In this paper, we investigate the mathematical reasoning
robustness in VLMs and evaluate how well these models perform under different
variants of the same question, such as changes in visual numerical values or
function graphs. While several vision-based math benchmarks have been developed
to assess VLMs' problem-solving capabilities, these benchmarks contain only
static sets of problems and cannot easily evaluate mathematical reasoning
robustness. To fill this gap, we introduce DynaMath, a dynamic visual math
benchmark designed for in-depth assessment of VLMs. DynaMath includes 501
high-quality, multi-topic seed questions, each represented as a Python program.
Those programs are carefully designed and annotated to enable the automatic
generation of a much larger set of concrete questions, including many different
types of visual and textual variations. DynaMath allows us to evaluate the
generalization ability of VLMs, by assessing their performance under varying
input conditions of a seed question. We evaluated 14 SOTA VLMs with 5,010
generated concrete questions. Our results show that the worst-case model
accuracy, defined as the percentage of correctly answered seed questions in all
10 variants, is significantly lower than the average-case accuracy. Our
analysis emphasizes the need to study the robustness of VLMs' reasoning
abilities, and DynaMath provides valuable insights to guide the development of
more reliable models for mathematical reasoning.


http://arxiv.org/abs/2410.20893
Evaluating the Robustness of LiDAR Point Cloud Tracking Against Adversarial Attack. (99%)
Shengjing Tian; Yinan Han; Xiantong Zhao; Bin Liu; Xiuping Liu
  In this study, we delve into the robustness of neural network-based LiDAR
point cloud tracking models under adversarial attacks, a critical aspect often
overlooked in favor of performance enhancement. These models, despite
incorporating advanced architectures like Transformer or Bird's Eye View (BEV),
tend to neglect robustness in the face of challenges such as adversarial
attacks, domain shifts, or data corruption. We instead focus on the robustness
of the tracking models under the threat of adversarial attacks. We begin by
establishing a unified framework for conducting adversarial attacks within the
context of 3D object tracking, which allows us to thoroughly investigate both
white-box and black-box attack strategies. For white-box attacks, we tailor
specific loss functions to accommodate various tracking paradigms and extend
existing methods such as FGSM, C\&W, and PGD to the point cloud domain. In
addressing black-box attack scenarios, we introduce a novel transfer-based
approach, the Target-aware Perturbation Generation (TAPG) algorithm, with the
dual objectives of achieving high attack performance and maintaining low
perceptibility. This method employs a heuristic strategy to enforce sparse
attack constraints and utilizes random sub-vector factorization to bolster
transferability. Our experimental findings reveal a significant vulnerability
in advanced tracking methods when subjected to both black-box and white-box
attacks, underscoring the necessity for incorporating robustness against
adversarial attacks into the design of LiDAR point cloud tracking models.
Notably, compared to existing methods, the TAPG also strikes an optimal balance
between the effectiveness of the attack and the concealment of the
perturbations.


http://arxiv.org/abs/2410.21471
AdvI2I: Adversarial Image Attack on Image-to-Image Diffusion models. (96%)
Yaopei Zeng; Yuanpu Cao; Bochuan Cao; Yurui Chang; Jinghui Chen; Lu Lin
  Recent advances in diffusion models have significantly enhanced the quality
of image synthesis, yet they have also introduced serious safety concerns,
particularly the generation of Not Safe for Work (NSFW) content. Previous
research has demonstrated that adversarial prompts can be used to generate NSFW
content. However, such adversarial text prompts are often easily detectable by
text-based filters, limiting their efficacy. In this paper, we expose a
previously overlooked vulnerability: adversarial image attacks targeting
Image-to-Image (I2I) diffusion models. We propose AdvI2I, a novel framework
that manipulates input images to induce diffusion models to generate NSFW
content. By optimizing a generator to craft adversarial images, AdvI2I
circumvents existing defense mechanisms, such as Safe Latent Diffusion (SLD),
without altering the text prompts. Furthermore, we introduce AdvI2I-Adaptive,
an enhanced version that adapts to potential countermeasures and minimizes the
resemblance between adversarial images and NSFW concept embeddings, making the
attack more resilient against defenses. Through extensive experiments, we
demonstrate that both AdvI2I and AdvI2I-Adaptive can effectively bypass current
safeguards, highlighting the urgent need for stronger security measures to
address the misuse of I2I diffusion models.


http://arxiv.org/abs/2410.20971
BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attacks. (93%)
Yunhan Zhao; Xiang Zheng; Lin Luo; Yige Li; Xingjun Ma; Yu-Gang Jiang
  Despite their superb multimodal capabilities, Vision-Language Models (VLMs)
have been shown to be vulnerable to jailbreak attacks, which are inference-time
attacks that induce the model to output harmful responses with tricky prompts.
It is thus essential to defend VLMs against potential jailbreaks for their
trustworthy deployment in real-world applications. In this work, we focus on
black-box defense for VLMs against jailbreak attacks. Existing black-box
defense methods are either unimodal or bimodal. Unimodal methods enhance either
the vision or language module of the VLM, while bimodal methods robustify the
model through text-image representation realignment. However, these methods
suffer from two limitations: 1) they fail to fully exploit the cross-modal
information, or 2) they degrade the model performance on benign inputs. To
address these limitations, we propose a novel blue-team method BlueSuffix that
defends the black-box target VLM against jailbreak attacks without compromising
its performance. BlueSuffix includes three key components: 1) a visual purifier
against jailbreak images, 2) a textual purifier against jailbreak texts, and 3)
a blue-team suffix generator fine-tuned via reinforcement learning for
enhancing cross-modal robustness. We empirically show on three VLMs (LLaVA,
MiniGPT-4, and Gemini) and two safety benchmarks (MM-SafetyBench and
RedTeam-2K) that BlueSuffix outperforms the baseline defenses by a significant
margin. Our BlueSuffix opens up a promising direction for defending VLMs
against jailbreak attacks.


http://arxiv.org/abs/2410.21492
FATH: Authentication-based Test-time Defense against Indirect Prompt Injection Attacks. (91%)
Jiongxiao Wang; Fangzhou Wu; Wendi Li; Jinsheng Pan; Edward Suh; Z. Morley Mao; Muhao Chen; Chaowei Xiao
  Large language models (LLMs) have been widely deployed as the backbone with
additional tools and text information for real-world applications. However,
integrating external information into LLM-integrated applications raises
significant security concerns. Among these, prompt injection attacks are
particularly threatening, where malicious instructions injected in the external
text information can exploit LLMs to generate answers as the attackers desire.
While both training-time and test-time defense methods have been developed to
mitigate such attacks, the unaffordable training costs associated with
training-time methods and the limited effectiveness of existing test-time
methods make them impractical. This paper introduces a novel test-time defense
strategy, named Formatting AuThentication with Hash-based tags (FATH). Unlike
existing approaches that prevent LLMs from answering additional instructions in
external text, our method implements an authentication system, requiring LLMs
to answer all received instructions with a security policy and selectively
filter out responses to user instructions as the final output. To achieve this,
we utilize hash-based authentication tags to label each response, facilitating
accurate identification of responses according to the user's instructions and
improving the robustness against adaptive attacks. Comprehensive experiments
demonstrate that our defense method can effectively defend against indirect
prompt injection attacks, achieving state-of-the-art performance under Llama3
and GPT3.5 models across various attack methods. Our code is released at:
https://github.com/Jayfeather1024/FATH


http://arxiv.org/abs/2410.21443
TACO: Adversarial Camouflage Optimization on Trucks to Fool Object Detectors. (88%)
Adonisz Dimitriu; Tamás Michaletzky; Viktor Remeli
  Adversarial attacks threaten the reliability of machine learning models in
critical applications like autonomous vehicles and defense systems. As object
detectors become more robust with models like YOLOv8, developing effective
adversarial methodologies is increasingly challenging. We present Truck
Adversarial Camouflage Optimization (TACO), a novel framework that generates
adversarial camouflage patterns on 3D vehicle models to deceive
state-of-the-art object detectors. Adopting Unreal Engine 5, TACO integrates
differentiable rendering with a Photorealistic Rendering Network to optimize
adversarial textures targeted at YOLOv8. To ensure the generated textures are
both effective in deceiving detectors and visually plausible, we introduce the
Convolutional Smooth Loss function, a generalized smooth loss function.
Experimental evaluations demonstrate that TACO significantly degrades YOLOv8's
detection performance, achieving an AP@0.5 of 0.0099 on unseen test data.
Furthermore, these adversarial patterns exhibit strong transferability to other
object detection models such as Faster R-CNN and earlier YOLO versions.


http://arxiv.org/abs/2410.20940
Attacking Misinformation Detection Using Adversarial Examples Generated by Language Models. (83%)
Piotr Przybyła
  We investigate the challenge of generating adversarial examples to test the
robustness of text classification algorithms detecting low-credibility content,
including propaganda, false claims, rumours and hyperpartisan news. We focus on
simulation of content moderation by setting realistic limits on the number of
queries an attacker is allowed to attempt. Within our solution (TREPAT),
initial rephrasings are generated by large language models with prompts
inspired by meaning-preserving NLP tasks, e.g. text simplification and style
transfer. Subsequently, these modifications are decomposed into small changes,
applied through beam search procedure until the victim classifier changes its
decision. The evaluation confirms the superiority of our approach in the
constrained scenario, especially in case of long input text (news articles),
where exhaustive search is not feasible.


http://arxiv.org/abs/2410.20911
Hacking Back the AI-Hacker: Prompt Injection as a Defense Against LLM-driven Cyberattacks. (80%)
Dario Pasquini; Evgenios M. Kornaropoulos; Giuseppe Ateniese
  Large language models (LLMs) are increasingly being harnessed to automate
cyberattacks, making sophisticated exploits more accessible and scalable. In
response, we propose a new defense strategy tailored to counter LLM-driven
cyberattacks. We introduce Mantis, a defensive framework that exploits LLMs'
susceptibility to adversarial inputs to undermine malicious operations. Upon
detecting an automated cyberattack, Mantis plants carefully crafted inputs into
system responses, leading the attacker's LLM to disrupt their own operations
(passive defense) or even compromise the attacker's machine (active defense).
By deploying purposefully vulnerable decoy services to attract the attacker and
using dynamic prompt injections for the attacker's LLM, Mantis can autonomously
hack back the attacker. In our experiments, Mantis consistently achieved over
95% effectiveness against automated LLM-driven attacks. To foster further
research and collaboration, Mantis is available as an open-source tool:
https://github.com/pasquini-dario/project_mantis


http://arxiv.org/abs/2410.21083
Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroring. (50%)
Honglin Mu; Han He; Yuxin Zhou; Yunlong Feng; Yang Xu; Libo Qin; Xiaoming Shi; Zeming Liu; Xudong Han; Qi Shi; Qingfu Zhu; Wanxiang Che
  Large language model (LLM) safety is a critical issue, with numerous studies
employing red team testing to enhance model security. Among these, jailbreak
methods explore potential vulnerabilities by crafting malicious prompts that
induce model outputs contrary to safety alignments. Existing black-box
jailbreak methods often rely on model feedback, repeatedly submitting queries
with detectable malicious instructions during the attack search process.
Although these approaches are effective, the attacks may be intercepted by
content moderators during the search process. We propose an improved transfer
attack method that guides malicious prompt construction by locally training a
mirror model of the target black-box model through benign data distillation.
This method offers enhanced stealth, as it does not involve submitting
identifiable malicious instructions to the target model during the search
phase. Our approach achieved a maximum attack success rate of 92%, or a
balanced value of 80% with an average of 1.5 detectable jailbreak queries per
sample against GPT-3.5 Turbo on a subset of AdvBench. These results underscore
the need for more robust defense mechanisms.


http://arxiv.org/abs/2410.20742
Mitigating Unauthorized Speech Synthesis for Voice Protection. (9%)
Zhisheng Zhang; Qianyi Yang; Derui Wang; Pengyang Huang; Yuxin Cao; Kai Ye; Jie Hao
  With just a few speech samples, it is possible to perfectly replicate a
speaker's voice in recent years, while malicious voice exploitation (e.g.,
telecom fraud for illegal financial gain) has brought huge hazards in our daily
lives. Therefore, it is crucial to protect publicly accessible speech data that
contains sensitive information, such as personal voiceprints. Most previous
defense methods have focused on spoofing speaker verification systems in timbre
similarity but the synthesized deepfake speech is still of high quality. In
response to the rising hazards, we devise an effective, transferable, and
robust proactive protection technology named Pivotal Objective Perturbation
(POP) that applies imperceptible error-minimizing noises on original speech
samples to prevent them from being effectively learned for text-to-speech (TTS)
synthesis models so that high-quality deepfake speeches cannot be generated. We
conduct extensive experiments on state-of-the-art (SOTA) TTS models utilizing
objective and subjective metrics to comprehensively evaluate our proposed
method. The experimental results demonstrate outstanding effectiveness and
transferability across various models. Compared to the speech unclarity score
of 21.94% from voice synthesizers trained on samples without protection,
POP-protected samples significantly increase it to 127.31%. Moreover, our
method shows robustness against noise reduction and data augmentation
techniques, thereby greatly reducing potential hazards.


http://arxiv.org/abs/2410.21146
Palisade -- Prompt Injection Detection Framework. (1%)
Sahasra Kokkula; Somanathan R; Nandavardhan R; Aashishkumar; G Divya
  The advent of Large Language Models LLMs marks a milestone in Artificial
Intelligence, altering how machines comprehend and generate human language.
However, LLMs are vulnerable to malicious prompt injection attacks, where
crafted inputs manipulate the models behavior in unintended ways, compromising
system integrity and causing incorrect outcomes. Conventional detection methods
rely on static, rule-based approaches, which often fail against sophisticated
threats like abnormal token sequences and alias substitutions, leading to
limited adaptability and higher rates of false positives and false
negatives.This paper proposes a novel NLP based approach for prompt injection
detection, emphasizing accuracy and optimization through a layered input
screening process. In this framework, prompts are filtered through three
distinct layers rule-based, ML classifier, and companion LLM before reaching
the target model, thereby minimizing the risk of malicious interaction.Tests
show the ML classifier achieves the highest accuracy among individual layers,
yet the multi-layer framework enhances overall detection accuracy by reducing
false negatives. Although this increases false positives, it minimizes the risk
of overlooking genuine injected prompts, thus prioritizing security.This
multi-layered detection approach highlights LLM vulnerabilities and provides a
comprehensive framework for future research, promoting secure interactions
between humans and AI systems.


http://arxiv.org/abs/2410.20432
Integrating uncertainty quantification into randomized smoothing based robustness guarantees. (98%)
Sina Däubener; Kira Maag; David Krueger; Asja Fischer
  Deep neural networks have proven to be extremely powerful, however, they are
also vulnerable to adversarial attacks which can cause hazardous incorrect
predictions in safety-critical applications. Certified robustness via
randomized smoothing gives a probabilistic guarantee that the smoothed
classifier's predictions will not change within an $\ell_2$-ball around a given
input. On the other hand (uncertainty) score-based rejection is a technique
often applied in practice to defend models against adversarial attacks. In this
work, we fuse these two approaches by integrating a classifier that abstains
from predicting when uncertainty is high into the certified robustness
framework. This allows us to derive two novel robustness guarantees for
uncertainty aware classifiers, namely (i) the radius of an $\ell_2$-ball around
the input in which the same label is predicted and uncertainty remains low and
(ii) the $\ell_2$-radius of a ball in which the predictions will either not
change or be uncertain. While the former provides robustness guarantees with
respect to attacks aiming at increased uncertainty, the latter informs about
the amount of input perturbation necessary to lead the uncertainty aware model
into a wrong prediction. Notably, this is on CIFAR10 up to 20.93% larger than
for models not allowing for uncertainty based rejection. We demonstrate, that
the novel framework allows for a systematic robustness evaluation of different
network architectures and uncertainty measures and to identify desired
properties of uncertainty quantification techniques. Moreover, we show that
leveraging uncertainty in a smoothed classifier helps out-of-distribution
detection.


http://arxiv.org/abs/2410.21330
LLM Robustness Against Misinformation in Biomedical Question Answering. (80%)
Alexander Bondarenko; Adrian Viehweger
  The retrieval-augmented generation (RAG) approach is used to reduce the
confabulation of large language models (LLMs) for question answering by
retrieving and providing additional context coming from external knowledge
sources (e.g., by adding the context to the prompt). However, injecting
incorrect information can mislead the LLM to generate an incorrect answer.
  In this paper, we evaluate the effectiveness and robustness of four LLMs
against misinformation - Gemma 2, GPT-4o-mini, Llama~3.1, and Mixtral - in
answering biomedical questions. We assess the answer accuracy on yes-no and
free-form questions in three scenarios: vanilla LLM answers (no context is
provided), "perfect" augmented generation (correct context is provided), and
prompt-injection attacks (incorrect context is provided). Our results show that
Llama 3.1 (70B parameters) achieves the highest accuracy in both vanilla
(0.651) and "perfect" RAG (0.802) scenarios. However, the accuracy gap between
the models almost disappears with "perfect" RAG, suggesting its potential to
mitigate the LLM's size-related effectiveness differences.
  We further evaluate the ability of the LLMs to generate malicious context on
one hand and the LLM's robustness against prompt-injection attacks on the other
hand, using metrics such as attack success rate (ASR), accuracy under attack,
and accuracy drop. As adversaries, we use the same four LLMs (Gemma 2,
GPT-4o-mini, Llama 3.1, and Mixtral) to generate incorrect context that is
injected in the target model's prompt. Interestingly, Llama is shown to be the
most effective adversary, causing accuracy drops of up to 0.48 for vanilla
answers and 0.63 for "perfect" RAG across target models. Our analysis reveals
that robustness rankings vary depending on the evaluation measure, highlighting
the complexity of assessing LLM resilience to adversarial attacks.


http://arxiv.org/abs/2410.21337
Fine-tuned Large Language Models (LLMs): Improved Prompt Injection Attacks Detection. (1%)
Md Abdur Rahman; Fan Wu; Alfredo Cuzzocrea; Sheikh Iqbal Ahamed
  Large language models (LLMs) are becoming a popular tool as they have
significantly advanced in their capability to tackle a wide range of
language-based tasks. However, LLMs applications are highly vulnerable to
prompt injection attacks, which poses a critical problem. These attacks target
LLMs applications through using carefully designed input prompts to divert the
model from adhering to original instruction, thereby it could execute
unintended actions. These manipulations pose serious security threats which
potentially results in data leaks, biased outputs, or harmful responses. This
project explores the security vulnerabilities in relation to prompt injection
attacks. To detect whether a prompt is vulnerable or not, we follows two
approaches: 1) a pre-trained LLM, and 2) a fine-tuned LLM. Then, we conduct a
thorough analysis and comparison of the classification performance. Firstly, we
use pre-trained XLM-RoBERTa model to detect prompt injections using test
dataset without any fine-tuning and evaluate it by zero-shot classification.
Then, this proposed work will apply supervised fine-tuning to this pre-trained
LLM using a task-specific labeled dataset from deepset in huggingface, and this
fine-tuned model achieves impressive results with 99.13\% accuracy, 100\%
precision, 98.33\% recall and 99.15\% F1-score thorough rigorous
experimentation and evaluation. We observe that our approach is highly
efficient in detecting prompt injection attacks.


http://arxiv.org/abs/2410.20103
Adversarial Attacks Against Double RIS-Assisted MIMO Systems-based Autoencoder in Finite-Scattering Environments. (99%)
Bui Duc Son; Ngo Nam Khanh; Chien Trinh Van; Dong In Kim
  Autoencoder permits the end-to-end optimization and design of wireless
communication systems to be more beneficial than traditional signal processing.
However, this emerging learning-based framework has weaknesses, especially
sensitivity to physical attacks. This paper explores adversarial attacks
against a double reconfigurable intelligent surface (RIS)-assisted
multiple-input and multiple-output (MIMO)-based autoencoder, where an adversary
employs encoded and decoded datasets to create adversarial perturbation and
fool the system. Because of the complex and dynamic data structures,
adversarial attacks are not unique, each having its own benefits. We,
therefore, propose three algorithms generating adversarial examples and
perturbations to attack the RIS-MIMO-based autoencoder, exploiting the gradient
descent and allowing for flexibility via varying the input dimensions.
Numerical results show that the proposed adversarial attack-based algorithm
significantly degrades the system performance regarding the symbol error rate
compared to the jamming attacks.


http://arxiv.org/abs/2410.20197
Transferable Adversarial Attacks on SAM and Its Downstream Models. (99%)
Song Xia; Wenhan Yang; Yi Yu; Xun Lin; Henghui Ding; Lingyu Duan; Xudong Jiang
  The utilization of large foundational models has a dilemma: while fine-tuning
downstream tasks from them holds promise for making use of the well-generalized
knowledge in practical applications, their open accessibility also poses
threats of adverse usage. This paper, for the first time, explores the
feasibility of adversarial attacking various downstream models fine-tuned from
the segment anything model (SAM), by solely utilizing the information from the
open-sourced SAM. In contrast to prevailing transfer-based adversarial attacks,
we demonstrate the existence of adversarial dangers even without accessing the
downstream task and dataset to train a similar surrogate model. To enhance the
effectiveness of the adversarial attack towards models fine-tuned on unknown
datasets, we propose a universal meta-initialization (UMI) algorithm to extract
the intrinsic vulnerability inherent in the foundation model, which is then
utilized as the prior knowledge to guide the generation of adversarial
perturbations. Moreover, by formulating the gradient difference in the
attacking process between the open-sourced SAM and its fine-tuned downstream
models, we theoretically demonstrate that a deviation occurs in the adversarial
update direction by directly maximizing the distance of encoded feature
embeddings in the open-sourced SAM. Consequently, we propose a gradient robust
loss that simulates the associated uncertainty with gradient-based noise
augmentation to enhance the robustness of generated adversarial examples (AEs)
towards this deviation, thus improving the transferability. Extensive
experiments demonstrate the effectiveness of the proposed universal
meta-initialized and gradient robust adversarial attack (UMI-GRAT) toward SAMs
and their downstream models. Code is available at
https://github.com/xiasong0501/GRAT.


http://arxiv.org/abs/2410.20097
Generative Adversarial Patches for Physical Attacks on Cross-Modal Pedestrian Re-Identification. (98%)
Yue Su; Hao Li; Maoguo Gong
  Visible-infrared pedestrian Re-identification (VI-ReID) aims to match
pedestrian images captured by infrared cameras and visible cameras. However,
VI-ReID, like other traditional cross-modal image matching tasks, poses
significant challenges due to its human-centered nature. This is evidenced by
the shortcomings of existing methods, which struggle to extract common features
across modalities, while losing valuable information when bridging the gap
between them in the implicit feature space, potentially compromising security.
To address this vulnerability, this paper introduces the first physical
adversarial attack against VI-ReID models. Our method, termed Edge-Attack,
specifically tests the models' ability to leverage deep-level implicit features
by focusing on edge information, the most salient explicit feature
differentiating individuals across modalities. Edge-Attack utilizes a novel
two-step approach. First, a multi-level edge feature extractor is trained in a
self-supervised manner to capture discriminative edge representations for each
individual. Second, a generative model based on Vision Transformer Generative
Adversarial Networks (ViTGAN) is employed to generate adversarial patches
conditioned on the extracted edge features. By applying these patches to
pedestrian clothing, we create realistic, physically-realizable adversarial
samples. This black-box, self-supervised approach ensures the generalizability
of our attack against various VI-ReID models. Extensive experiments on
SYSU-MM01 and RegDB datasets, including real-world deployments, demonstrate the
effectiveness of Edge- Attack in significantly degrading the performance of
state-of-the-art VI-ReID methods.


http://arxiv.org/abs/2410.20136
CodePurify: Defend Backdoor Attacks on Neural Code Models via Entropy-based Purification. (76%)
Fangwen Mu; Junjie Wang; Zhuohao Yu; Lin Shi; Song Wang; Mingyang Li; Qing Wang
  Neural code models have found widespread success in tasks pertaining to code
intelligence, yet they are vulnerable to backdoor attacks, where an adversary
can manipulate the victim model's behavior by inserting triggers into the
source code. Recent studies indicate that advanced backdoor attacks can achieve
nearly 100% attack success rates on many software engineering tasks. However,
effective defense techniques against such attacks remain insufficiently
explored. In this study, we propose CodePurify, a novel defense against
backdoor attacks on code models through entropy-based purification.
Entropy-based purification involves the process of precisely detecting and
eliminating the possible triggers in the source code while preserving its
semantic information. Within this process, CodePurify first develops a
confidence-driven entropy-based measurement to determine whether a code snippet
is poisoned and, if so, locates the triggers. Subsequently, it purifies the
code by substituting the triggers with benign tokens using a masked language
model. We extensively evaluate CodePurify against four advanced backdoor
attacks across three representative tasks and two popular code models. The
results show that CodePurify significantly outperforms four commonly used
defense baselines, improving average defense performance by at least 40%, 40%,
and 12% across the three tasks, respectively. These findings highlight the
potential of CodePurify to serve as a robust defense against backdoor attacks
on neural code models.


http://arxiv.org/abs/2410.20250
Robust Model Evaluation over Large-scale Federated Networks. (2%)
Amir Najafi; Samin Mahdizadeh Sani; Farzan Farnia
  In this paper, we address the challenge of certifying the performance of a
machine learning model on an unseen target network, using measurements from an
available source network. We focus on a scenario where heterogeneous datasets
are distributed across a source network of clients, all connected to a central
server. Specifically, consider a source network "A" composed of $K$ clients,
each holding private data from unique and heterogeneous distributions, which
are assumed to be independent samples from a broader meta-distribution $\mu$.
Our goal is to provide certified guarantees for the model's performance on a
different, unseen target network "B," governed by another meta-distribution
$\mu'$, assuming the deviation between $\mu$ and $\mu'$ is bounded by either
the Wasserstein distance or an $f$-divergence. We derive theoretical guarantees
for the model's empirical average loss and provide uniform bounds on the risk
CDF, where the latter correspond to novel and adversarially robust versions of
the Glivenko-Cantelli theorem and the Dvoretzky-Kiefer-Wolfowitz (DKW)
inequality. Our bounds are computable in polynomial time with a polynomial
number of queries to the $K$ clients, preserving client privacy by querying
only the model's (potentially adversarial) loss on private data. We also
establish non-asymptotic generalization bounds that consistently converge to
zero as both $K$ and the minimum client sample size grow. Extensive empirical
evaluations validate the robustness and practicality of our bounds across
real-world tasks.


http://arxiv.org/abs/2410.21276
GPT-4o System Card. (76%)
Tony OpenAI; Tony :; Aaron Tony Hurst; Adam Tony Lerer; Adam P. Tony Goucher; Adam Tony Perelman; Aditya Tony Ramesh; Aidan Tony Clark; AJ Tony Ostrow; Akila Tony Welihinda; Alan Tony Hayes; Alec Tony Radford; Aleksander Tony Mądry; Alex Tony Baker-Whitcomb; Alex Tony Beutel; Alex Tony Borzunov; Alex Tony Carney; Alex Tony Chow; Alex Tony Kirillov; Alex Tony Nichol; Alex Tony Paino; Alex Tony Renzin; Alex Tachard Tony Passos; Alexander Tony Kirillov; Alexi Tony Christakis; Alexis Tony Conneau; Ali Tony Kamali; Allan Tony Jabri; Allison Tony Moyer; Allison Tony Tam; Amadou Tony Crookes; Amin Tony Tootoochian; Amin Tony Tootoonchian; Ananya Tony Kumar; Andrea Tony Vallone; Andrej Tony Karpathy; Andrew Tony Braunstein; Andrew Tony Cann; Andrew Tony Codispoti; Andrew Tony Galu; Andrew Tony Kondrich; Andrew Tony Tulloch; Andrey Tony Mishchenko; Angela Tony Baek; Angela Tony Jiang; Antoine Tony Pelisse; Antonia Tony Woodford; Anuj Tony Gosalia; Arka Tony Dhar; Ashley Tony Pantuliano; Avi Tony Nayak; Avital Tony Oliver; Barret Tony Zoph; Behrooz Tony Ghorbani; Ben Tony Leimberger; Ben Tony Rossen; Ben Tony Sokolowsky; Ben Tony Wang; Benjamin Tony Zweig; Beth Tony Hoover; Blake Tony Samic; Bob Tony McGrew; Bobby Tony Spero; Bogo Tony Giertler; Bowen Tony Cheng; Brad Tony Lightcap; Brandon Tony Walkin; Brendan Tony Quinn; Brian Tony Guarraci; Brian Tony Hsu; Bright Tony Kellogg; Brydon Tony Eastman; Camillo Tony Lugaresi; Carroll Tony Wainwright; Cary Tony Bassin; Cary Tony Hudson; Casey Tony Chu; Chad Tony Nelson; Chak Tony Li; Chan Jun Tony Shern; Channing Tony Conger; Charlotte Tony Barette; Chelsea Tony Voss; Chen Tony Ding; Cheng Tony Lu; Chong Tony Zhang; Chris Tony Beaumont; Chris Tony Hallacy; Chris Tony Koch; Christian Tony Gibson; Christina Tony Kim; Christine Tony Choi; Christine Tony McLeavey; Christopher Tony Hesse; Claudia Tony Fischer; Clemens Tony Winter; Coley Tony Czarnecki; Colin Tony Jarvis; Colin Tony Wei; Constantin Tony Koumouzelis; Dane Tony Sherburn; Daniel Tony Kappler; Daniel Tony Levin; Daniel Tony Levy; David Tony Carr; David Tony Farhi; David Tony Mely; David Tony Robinson; David Tony Sasaki; Denny Tony Jin; Dev Tony Valladares; Dimitris Tony Tsipras; Doug Tony Li; Duc Phong Tony Nguyen; Duncan Tony Findlay; Edede Tony Oiwoh; Edmund Tony Wong; Ehsan Tony Asdar; Elizabeth Tony Proehl; Elizabeth Tony Yang; Eric Tony Antonow; Eric Tony Kramer; Eric Tony Peterson; Eric Tony Sigler; Eric Tony Wallace; Eugene Tony Brevdo; Evan Tony Mays; Farzad Tony Khorasani; Felipe Petroski Tony Such; Filippo Tony Raso; Francis Tony Zhang; Lohmann Fred Tony von; Freddie Tony Sulit; Gabriel Tony Goh; Gene Tony Oden; Geoff Tony Salmon; Giulio Tony Starace; Greg Tony Brockman; Hadi Tony Salman; Haiming Tony Bao; Haitang Tony Hu; Hannah Tony Wong; Haoyu Tony Wang; Heather Tony Schmidt; Heather Tony Whitney; Heewoo Tony Jun; Hendrik Tony Kirchner; Henrique Ponde de Oliveira Tony Pinto; Hongyu Tony Ren; Huiwen Tony Chang; Hyung Won Tony Chung; Ian Tony Kivlichan; Ian Tony O'Connell; Ian Tony O'Connell; Ian Tony Osband; Ian Tony Silber; Ian Tony Sohl; Ibrahim Tony Okuyucu; Ikai Tony Lan; Ilya Tony Kostrikov; Ilya Tony Sutskever; Ingmar Tony Kanitscheider; Ishaan Tony Gulrajani; Jacob Tony Coxon; Jacob Tony Menick; Jakub Tony Pachocki; James Tony Aung; James Tony Betker; James Tony Crooks; James Tony Lennon; Jamie Tony Kiros; Jan Tony Leike; Jane Tony Park; Jason Tony Kwon; Jason Tony Phang; Jason Tony Teplitz; Jason Tony Wei; Jason Tony Wolfe; Jay Tony Chen; Jeff Tony Harris; Jenia Tony Varavva; Jessica Gan Tony Lee; Jessica Tony Shieh; Ji Tony Lin; Jiahui Tony Yu; Jiayi Tony Weng; Jie Tony Tang; Jieqi Tony Yu; Joanne Tony Jang; Joaquin Quinonero Tony Candela; Joe Tony Beutler; Joe Tony Landers; Joel Tony Parish; Johannes Tony Heidecke; John Tony Schulman; Jonathan Tony Lachman; Jonathan Tony McKay; Jonathan Tony Uesato; Jonathan Tony Ward; Jong Wook Tony Kim; Joost Tony Huizinga; Jordan Tony Sitkin; Jos Tony Kraaijeveld; Josh Tony Gross; Josh Tony Kaplan; Josh Tony Snyder; Joshua Tony Achiam; Joy Tony Jiao; Joyce Tony Lee; Juntang Tony Zhuang; Justyn Tony Harriman; Kai Tony Fricke; Kai Tony Hayashi; Karan Tony Singhal; Katy Tony Shi; Kavin Tony Karthik; Kayla Tony Wood; Kendra Tony Rimbach; Kenny Tony Hsu; Kenny Tony Nguyen; Keren Tony Gu-Lemberg; Kevin Tony Button; Kevin Tony Liu; Kiel Tony Howe; Krithika Tony Muthukumar; Kyle Tony Luther; Lama Tony Ahmad; Larry Tony Kai; Lauren Tony Itow; Lauren Tony Workman; Leher Tony Pathak; Leo Tony Chen; Li Tony Jing; Lia Tony Guy; Liam Tony Fedus; Liang Tony Zhou; Lien Tony Mamitsuka; Lilian Tony Weng; Lindsay Tony McCallum; Lindsey Tony Held; Long Tony Ouyang; Louis Tony Feuvrier; Lu Tony Zhang; Lukas Tony Kondraciuk; Lukasz Tony Kaiser; Luke Tony Hewitt; Luke Tony Metz; Lyric Tony Doshi; Mada Tony Aflak; Maddie Tony Simens; Madelaine Tony Boyd; Madeleine Tony Thompson; Marat Tony Dukhan; Mark Tony Chen; Mark Tony Gray; Mark Tony Hudnall; Marvin Tony Zhang; Marwan Tony Aljubeh; Mateusz Tony Litwin; Matthew Tony Zeng; Max Tony Johnson; Maya Tony Shetty; Mayank Tony Gupta; Meghan Tony Shah; Mehmet Tony Yatbaz; Meng Jia Tony Yang; Mengchao Tony Zhong; Mia Tony Glaese; Mianna Tony Chen; Michael Tony Janner; Michael Tony Lampe; Michael Tony Petrov; Michael Tony Wu; Michele Tony Wang; Michelle Tony Fradin; Michelle Tony Pokrass; Miguel Tony Castro; Castro Miguel Oom Temudo Tony de; Mikhail Tony Pavlov; Miles Tony Brundage; Miles Tony Wang; Minal Tony Khan; Mira Tony Murati; Mo Tony Bavarian; Molly Tony Lin; Murat Tony Yesildal; Nacho Tony Soto; Natalia Tony Gimelshein; Natalie Tony Cone; Natalie Tony Staudacher; Natalie Tony Summers; Natan Tony LaFontaine; Neil Tony Chowdhury; Nick Tony Ryder; Nick Tony Stathas; Nick Tony Turley; Nik Tony Tezak; Niko Tony Felix; Nithanth Tony Kudige; Nitish Tony Keskar; Noah Tony Deutsch; Noel Tony Bundick; Nora Tony Puckett; Ofir Tony Nachum; Ola Tony Okelola; Oleg Tony Boiko; Oleg Tony Murk; Oliver Tony Jaffe; Olivia Tony Watkins; Olivier Tony Godement; Owen Tony Campbell-Moore; Patrick Tony Chao; Paul Tony McMillan; Pavel Tony Belov; Peng Tony Su; Peter Tony Bak; Peter Tony Bakkum; Peter Tony Deng; Peter Tony Dolan; Peter Tony Hoeschele; Peter Tony Welinder; Phil Tony Tillet; Philip Tony Pronin; Philippe Tony Tillet; Prafulla Tony Dhariwal; Qiming Tony Yuan; Rachel Tony Dias; Rachel Tony Lim; Rahul Tony Arora; Rajan Tony Troll; Randall Tony Lin; Rapha Gontijo Tony Lopes; Raul Tony Puri; Reah Tony Miyara; Reimar Tony Leike; Renaud Tony Gaubert; Reza Tony Zamani; Ricky Tony Wang; Rob Tony Donnelly; Rob Tony Honsby; Rocky Tony Smith; Rohan Tony Sahai; Rohit Tony Ramchandani; Romain Tony Huet; Rory Tony Carmichael; Rowan Tony Zellers; Roy Tony Chen; Ruby Tony Chen; Ruslan Tony Nigmatullin; Ryan Tony Cheu; Saachi Tony Jain; Sam Tony Altman; Sam Tony Schoenholz; Sam Tony Toizer; Samuel Tony Miserendino; Sandhini Tony Agarwal; Sara Tony Culver; Scott Tony Ethersmith; Scott Tony Gray; Sean Tony Grove; Sean Tony Metzger; Shamez Tony Hermani; Shantanu Tony Jain; Shengjia Tony Zhao; Sherwin Tony Wu; Shino Tony Jomoto; Shirong Tony Wu; Tony Shuaiqi; Xia; Sonia Phene; Spencer Papay; Srinivas Narayanan; Steve Coffey; Steve Lee; Stewart Hall; Suchir Balaji; Tal Broda; Tal Stramer; Tao Xu; Tarun Gogineni; Taya Christianson; Ted Sanders; Tejal Patwardhan; Thomas Cunninghman; Thomas Degry; Thomas Dimson; Thomas Raoux; Thomas Shadwell; Tianhao Zheng; Todd Underwood; Todor Markov; Toki Sherbakov; Tom Rubin; Tom Stasi; Tomer Kaftan; Tristan Heywood; Troy Peterson; Tyce Walters; Tyna Eloundou; Valerie Qi; Veit Moeller; Vinnie Monaco; Vishal Kuo; Vlad Fomenko; Wayne Chang; Weiyi Zheng; Wenda Zhou; Wesam Manassra; Will Sheu; Wojciech Zaremba; Yash Patil; Yilei Qian; Yongjik Kim; Youlong Cheng; Yu Zhang; Yuchen He; Yuchen Zhang; Yujia Jin; Yunxing Dai; Yury Malkov
  GPT-4o is an autoregressive omni model that accepts as input any combination
of text, audio, image, and video, and generates any combination of text, audio,
and image outputs. It's trained end-to-end across text, vision, and audio,
meaning all inputs and outputs are processed by the same neural network. GPT-4o
can respond to audio inputs in as little as 232 milliseconds, with an average
of 320 milliseconds, which is similar to human response time in conversation.
It matches GPT-4 Turbo performance on text in English and code, with
significant improvement on text in non-English languages, while also being much
faster and 50\% cheaper in the API. GPT-4o is especially better at vision and
audio understanding compared to existing models. In line with our commitment to
building AI safely and consistent with our voluntary commitments to the White
House, we are sharing the GPT-4o System Card, which includes our Preparedness
Framework evaluations. In this System Card, we provide a detailed look at
GPT-4o's capabilities, limitations, and safety evaluations across multiple
categories, focusing on speech-to-speech while also evaluating text and image
capabilities, and measures we've implemented to ensure the model is safe and
aligned. We also include third-party assessments on dangerous capabilities, as
well as discussion of potential societal impacts of GPT-4o's text and vision
capabilities.


http://arxiv.org/abs/2410.19937
RobustKV: Defending Large Language Models against Jailbreak Attacks via KV Eviction. (64%)
Tanqiu Jiang; Zian Wang; Jiacheng Liang; Changjiang Li; Yuhui Wang; Ting Wang
  Jailbreak attacks circumvent LLMs' built-in safeguards by concealing harmful
queries within jailbreak prompts. While existing defenses primarily focus on
mitigating the effects of jailbreak prompts, they often prove inadequate as
jailbreak prompts can take arbitrary, adaptive forms. This paper presents
RobustKV, a novel defense that adopts a fundamentally different approach by
selectively removing critical tokens of harmful queries from key-value (KV)
caches. Intuitively, for a jailbreak prompt to be effective, its tokens must
achieve sufficient `importance' (as measured by attention scores), which
inevitably lowers the importance of tokens in the concealed harmful query.
Thus, by strategically evicting the KVs of the lowest-ranked tokens, RobustKV
diminishes the presence of the harmful query in the KV cache, thus preventing
the LLM from generating malicious responses. Extensive evaluation using
benchmark datasets and models demonstrates that RobustKV effectively counters
state-of-the-art jailbreak attacks while maintaining the LLM's general
performance on benign queries. Moreover, RobustKV creates an intriguing
evasiveness dilemma for adversaries, forcing them to balance between evading
RobustKV and bypassing the LLM's built-in safeguards. This trade-off
contributes to RobustKV's robustness against adaptive attacks. (warning: this
paper contains potentially harmful content generated by LLMs.)


http://arxiv.org/abs/2410.20019
Attacks against Abstractive Text Summarization Models through Lead Bias and Influence Functions. (62%)
Poojitha Thota; Shirin Nilizadeh
  Large Language Models have introduced novel opportunities for text
comprehension and generation. Yet, they are vulnerable to adversarial
perturbations and data poisoning attacks, particularly in tasks like text
classification and translation. However, the adversarial robustness of
abstractive text summarization models remains less explored. In this work, we
unveil a novel approach by exploiting the inherent lead bias in summarization
models, to perform adversarial perturbations. Furthermore, we introduce an
innovative application of influence functions, to execute data poisoning, which
compromises the model's integrity. This approach not only shows a skew in the
models behavior to produce desired outcomes but also shows a new behavioral
change, where models under attack tend to generate extractive summaries rather
than abstractive summaries.


http://arxiv.org/abs/2410.19427
Expose Before You Defend: Unifying and Enhancing Backdoor Defenses via Exposed Models. (56%)
Yige Li; Hanxun Huang; Jiaming Zhang; Xingjun Ma; Yu-Gang Jiang
  Backdoor attacks covertly implant triggers into deep neural networks (DNNs)
by poisoning a small portion of the training data with pre-designed backdoor
triggers. This vulnerability is exacerbated in the era of large models, where
extensive (pre-)training on web-crawled datasets is susceptible to compromise.
In this paper, we introduce a novel two-step defense framework named Expose
Before You Defend (EBYD). EBYD unifies existing backdoor defense methods into a
comprehensive defense system with enhanced performance. Specifically, EBYD
first exposes the backdoor functionality in the backdoored model through a
model preprocessing step called backdoor exposure, and then applies detection
and removal methods to the exposed model to identify and eliminate the backdoor
features. In the first step of backdoor exposure, we propose a novel technique
called Clean Unlearning (CUL), which proactively unlearns clean features from
the backdoored model to reveal the hidden backdoor features. We also explore
various model editing/modification techniques for backdoor exposure, including
fine-tuning, model sparsification, and weight perturbation. Using EBYD, we
conduct extensive experiments on 10 image attacks and 6 text attacks across 2
vision datasets (CIFAR-10 and an ImageNet subset) and 4 language datasets
(SST-2, IMDB, Twitter, and AG's News). The results demonstrate the importance
of backdoor exposure for backdoor defense, showing that the exposed models can
significantly benefit a range of downstream defense tasks, including backdoor
label detection, backdoor trigger recovery, backdoor model detection, and
backdoor removal. We hope our work could inspire more research in developing
advanced defense frameworks with exposed models. Our code is available at:
https://github.com/bboylyg/Expose-Before-You-Defend.


http://arxiv.org/abs/2410.20026
Towards Robust Algorithms for Surgical Phase Recognition via Digital Twin-based Scene Representation. (2%)
Hao Ding; Yuqian Zhang; Hongchao Shu; Xu Lian; Ji Woong Kim; Axel Krieger; Mathias Unberath
  Purpose: Surgical phase recognition (SPR) is an integral component of
surgical data science, enabling high-level surgical analysis. End-to-end
trained neural networks that predict surgical phase directly from videos have
shown excellent performance on benchmarks. However, these models struggle with
robustness due to non-causal associations in the training set, resulting in
poor generalizability. Our goal is to improve model robustness to variations in
the surgical videos by leveraging the digital twin (DT) paradigm -- an
intermediary layer to separate high-level analysis (SPR) from low-level
processing (geometric understanding). This approach takes advantage of the
recent vision foundation models that ensure reliable low-level scene
understanding to craft DT-based scene representations that support various
high-level tasks.
  Methods: We present a DT-based framework for SPR from videos. The framework
employs vision foundation models to extract representations. We embed the
representation in place of raw video inputs in the state-of-the-art Surgformer
model. The framework is trained on the Cholec80 dataset and evaluated on
out-of-distribution (OOD) and corrupted test samples.
  Results: Contrary to the vulnerability of the baseline model, our framework
demonstrates strong robustness on both OOD and corrupted samples, with a
video-level accuracy of 51.1 on the challenging CRCD dataset, 96.0 on an
internal robotics training dataset, and 64.4 on a highly corrupted Cholec80
test set.
  Conclusion: Our findings lend support to the thesis that DT-based scene
representations are effective in enhancing model robustness. Future work will
seek to improve the feature informativeness, automate feature extraction, and
incorporate interpretability for a more comprehensive framework.


http://arxiv.org/abs/2410.18648
GADT: Enhancing Transferable Adversarial Attacks through Gradient-guided Adversarial Data Transformation. (99%)
Yating Ma; Xiaogang Xu; Liming Fang; Zhe Liu
  Current Transferable Adversarial Examples (TAE) are primarily generated by
adding Adversarial Noise (AN). Recent studies emphasize the importance of
optimizing Data Augmentation (DA) parameters along with AN, which poses a
greater threat to real-world AI applications. However, existing DA-based
strategies often struggle to find optimal solutions due to the challenging DA
search procedure without proper guidance. In this work, we propose a novel
DA-based attack algorithm, GADT. GADT identifies suitable DA parameters through
iterative antagonism and uses posterior estimates to update AN based on these
parameters. We uniquely employ a differentiable DA operation library to
identify adversarial DA parameters and introduce a new loss function as a
metric during DA optimization. This loss term enhances adversarial effects
while preserving the original image content, maintaining attack crypticity.
Extensive experiments on public datasets with various networks demonstrate that
GADT can be integrated with existing transferable attack methods, updating
their DA parameters effectively while retaining their AN formulation
strategies. Furthermore, GADT can be utilized in other black-box attack
scenarios, e.g., query-based attacks, offering a new avenue to enhance attacks
on real-world AI applications in both research and industrial contexts.


http://arxiv.org/abs/2410.19160
Adversarial Attacks on Large Language Models Using Regularized Relaxation. (98%)
Samuel Jacob Chacko; Sajib Biswas; Chashi Mahiul Islam; Fatema Tabassum Liza; Xiuwen Liu
  As powerful Large Language Models (LLMs) are now widely used for numerous
practical applications, their safety is of critical importance. While alignment
techniques have significantly improved overall safety, LLMs remain vulnerable
to carefully crafted adversarial inputs. Consequently, adversarial attack
methods are extensively used to study and understand these vulnerabilities.
However, current attack methods face significant limitations. Those relying on
optimizing discrete tokens suffer from limited efficiency, while continuous
optimization techniques fail to generate valid tokens from the model's
vocabulary, rendering them impractical for real-world applications. In this
paper, we propose a novel technique for adversarial attacks that overcomes
these limitations by leveraging regularized gradients with continuous
optimization methods. Our approach is two orders of magnitude faster than the
state-of-the-art greedy coordinate gradient-based method, significantly
improving the attack success rate on aligned language models. Moreover, it
generates valid tokens, addressing a fundamental limitation of existing
continuous optimization methods. We demonstrate the effectiveness of our attack
on five state-of-the-art LLMs using four datasets.


http://arxiv.org/abs/2410.18469
Iterative Self-Tuning LLMs for Enhanced Jailbreaking Capabilities. (88%)
Chung-En Sun; Xiaodong Liu; Weiwei Yang; Tsui-Wei Weng; Hao Cheng; Aidan San; Michel Galley; Jianfeng Gao
  Recent research has shown that Large Language Models (LLMs) are vulnerable to
automated jailbreak attacks, where adversarial suffixes crafted by algorithms
appended to harmful queries bypass safety alignment and trigger unintended
responses. Current methods for generating these suffixes are computationally
expensive and have low Attack Success Rates (ASR), especially against
well-aligned models like Llama2 and Llama3. To overcome these limitations, we
introduce ADV-LLM, an iterative self-tuning process that crafts adversarial
LLMs with enhanced jailbreak ability. Our framework significantly reduces the
computational cost of generating adversarial suffixes while achieving nearly
100\% ASR on various open-source LLMs. Moreover, it exhibits strong attack
transferability to closed-source models, achieving 99% ASR on GPT-3.5 and 49%
ASR on GPT-4, despite being optimized solely on Llama3. Beyond improving
jailbreak ability, ADV-LLM provides valuable insights for future safety
alignment research through its ability to generate large datasets for studying
LLM safety.


http://arxiv.org/abs/2410.19230
Humanizing the Machine: Proxy Attacks to Mislead LLM Detectors. (68%)
Tianchun Wang; Yuanzhou Chen; Zichuan Liu; Zhanwen Chen; Haifeng Chen; Xiang Zhang; Wei Cheng
  The advent of large language models (LLMs) has revolutionized the field of
text generation, producing outputs that closely mimic human-like writing.
Although academic and industrial institutions have developed detectors to
prevent the malicious usage of LLM-generated texts, other research has doubt
about the robustness of these systems. To stress test these detectors, we
introduce a proxy-attack strategy that effortlessly compromises LLMs, causing
them to produce outputs that align with human-written text and mislead
detection systems. Our method attacks the source model by leveraging a
reinforcement learning (RL) fine-tuned humanized small language model (SLM) in
the decoding phase. Through an in-depth analysis, we demonstrate that our
attack strategy is capable of generating responses that are indistinguishable
to detectors, preventing them from differentiating between machine-generated
and human-written text. We conduct systematic evaluations on extensive datasets
using proxy-attacked open-source models, including Llama2-13B, Llama3-70B, and
Mixtral-8*7B in both white- and black-box settings. Our findings show that the
proxy-attack strategy effectively deceives the leading detectors, resulting in
an average AUROC drop of 70.4% across multiple datasets, with a maximum drop of
90.3% on a single dataset. Furthermore, in cross-discipline scenarios, our
strategy also bypasses these detectors, leading to a significant relative
decrease of up to 90.9%, while in cross-language scenario, the drop reaches
91.3%. Despite our proxy-attack strategy successfully bypassing the detectors
with such significant relative drops, we find that the generation quality of
the attacked models remains preserved, even within a modest utility budget,
when compared to the text produced by the original, unattacked source model.


http://arxiv.org/abs/2410.18556
Complexity Matters: Effective Dimensionality as a Measure for Adversarial Robustness. (33%)
David Khachaturov; Robert Mullins
  Quantifying robustness in a single measure for the purposes of model
selection, development of adversarial training methods, and anticipating trends
has so far been elusive. The simplest metric to consider is the number of
trainable parameters in a model but this has previously been shown to be
insufficient at explaining robustness properties. A variety of other metrics,
such as ones based on boundary thickness and gradient flatness have been
proposed but have been shown to be inadequate proxies for robustness.
  In this work, we investigate the relationship between a model's effective
dimensionality, which can be thought of as model complexity, and its robustness
properties. We run experiments on commercial-scale models that are often used
in real-world environments such as YOLO and ResNet. We reveal a near-linear
inverse relationship between effective dimensionality and adversarial
robustness, that is models with a lower dimensionality exhibit better
robustness. We investigate the effect of a variety of adversarial training
methods on effective dimensionality and find the same inverse linear
relationship present, suggesting that effective dimensionality can serve as a
useful criterion for model selection and robustness evaluation, providing a
more nuanced and effective metric than parameter count or previously-tested
measures.


http://arxiv.org/abs/2410.18775
Robust Watermarking Using Generative Priors Against Image Editing: From Benchmarking to Advances. (11%)
Shilin Lu; Zihan Zhou; Jiayou Lu; Yuanzhi Zhu; Adams Wai-Kin Kong
  Current image watermarking methods are vulnerable to advanced image editing
techniques enabled by large-scale text-to-image models. These models can
distort embedded watermarks during editing, posing significant challenges to
copyright protection. In this work, we introduce W-Bench, the first
comprehensive benchmark designed to evaluate the robustness of watermarking
methods against a wide range of image editing techniques, including image
regeneration, global editing, local editing, and image-to-video generation.
Through extensive evaluations of eleven representative watermarking methods
against prevalent editing techniques, we demonstrate that most methods fail to
detect watermarks after such edits. To address this limitation, we propose
VINE, a watermarking method that significantly enhances robustness against
various image editing techniques while maintaining high image quality. Our
approach involves two key innovations: (1) we analyze the frequency
characteristics of image editing and identify that blurring distortions exhibit
similar frequency properties, which allows us to use them as surrogate attacks
during training to bolster watermark robustness; (2) we leverage a large-scale
pretrained diffusion model SDXL-Turbo, adapting it for the watermarking task to
achieve more imperceptible and robust watermark embedding. Experimental results
show that our method achieves outstanding watermarking performance under
various image editing techniques, outperforming existing methods in both image
quality and robustness. Code is available at https://github.com/Shilin-LU/VINE.


http://arxiv.org/abs/2410.18215
Advancing NLP Security by Leveraging LLMs as Adversarial Engines. (98%)
Sudarshan Srinivasan; Maria Mahbub; Amir Sadovnik
  This position paper proposes a novel approach to advancing NLP security by
leveraging Large Language Models (LLMs) as engines for generating diverse
adversarial attacks. Building upon recent work demonstrating LLMs'
effectiveness in creating word-level adversarial examples, we argue for
expanding this concept to encompass a broader range of attack types, including
adversarial patches, universal perturbations, and targeted attacks. We posit
that LLMs' sophisticated language understanding and generation capabilities can
produce more effective, semantically coherent, and human-like adversarial
examples across various domains and classifier architectures. This paradigm
shift in adversarial NLP has far-reaching implications, potentially enhancing
model robustness, uncovering new vulnerabilities, and driving innovation in
defense mechanisms. By exploring this new frontier, we aim to contribute to the
development of more secure, reliable, and trustworthy NLP systems for critical
applications.


http://arxiv.org/abs/2410.18267
Backdoor in Seconds: Unlocking Vulnerabilities in Large Pre-trained Models via Model Editing. (93%)
Dongliang Guo; Mengxuan Hu; Zihan Guan; Junfeng Guo; Thomas Hartvigsen; Sheng Li
  Large pre-trained models have achieved notable success across a range of
downstream tasks. However, recent research shows that a type of adversarial
attack ($\textit{i.e.,}$ backdoor attack) can manipulate the behavior of
machine learning models through contaminating their training dataset, posing
significant threat in the real-world application of large pre-trained model,
especially for those customized models. Therefore, addressing the unique
challenges for exploring vulnerability of pre-trained models is of paramount
importance. Through empirical studies on the capability for performing backdoor
attack in large pre-trained models ($\textit{e.g.,}$ ViT), we find the
following unique challenges of attacking large pre-trained models: 1) the
inability to manipulate or even access large training datasets, and 2) the
substantial computational resources required for training or fine-tuning these
models. To address these challenges, we establish new standards for an
effective and feasible backdoor attack in the context of large pre-trained
models. In line with these standards, we introduce our EDT model, an
\textbf{E}fficient, \textbf{D}ata-free, \textbf{T}raining-free backdoor attack
method. Inspired by model editing techniques, EDT injects an editing-based
lightweight codebook into the backdoor of large pre-trained models, which
replaces the embedding of the poisoned image with the target image without
poisoning the training dataset or training the victim model. Our experiments,
conducted across various pre-trained models such as ViT, CLIP, BLIP, and stable
diffusion, and on downstream tasks including image classification, image
captioning, and image generation, demonstrate the effectiveness of our method.
Our code is available in the supplementary material.


http://arxiv.org/abs/2410.19863
Breaking the Illusion: Real-world Challenges for Adversarial Patches in Object Detection. (70%)
Jakob Shack; Katarina Petrovic; Olga Saukh
  Adversarial attacks pose a significant threat to the robustness and
reliability of machine learning systems, particularly in computer vision
applications. This study investigates the performance of adversarial patches
for the YOLO object detection network in the physical world. Two attacks were
tested: a patch designed to be placed anywhere within the scene - global patch,
and another patch intended to partially overlap with specific object targeted
for removal from detection - local patch. Various factors such as patch size,
position, rotation, brightness, and hue were analyzed to understand their
impact on the effectiveness of the adversarial patches. The results reveal a
notable dependency on these parameters, highlighting the challenges in
maintaining attack efficacy in real-world conditions. Learning to align
digitally applied transformation parameters with those measured in the real
world still results in up to a 64\% discrepancy in patch performance. These
findings underscore the importance of understanding environmental influences on
adversarial attacks, which can inform the development of more robust defenses
for practical machine learning applications.


http://arxiv.org/abs/2410.17910
Slot: Provenance-Driven APT Detection through Graph Reinforcement Learning. (16%)
Wei Qiao; Yebo Feng; Teng Li; Zhuo Ma; Yulong Shen; JianFeng Ma; Yang Liu
  Advanced Persistent Threats (APTs) represent sophisticated cyberattacks
characterized by their ability to remain undetected within the victim system
for extended periods, aiming to exfiltrate sensitive data or disrupt
operations. Existing detection approaches often struggle to effectively
identify these complex threats, construct the attack chain for defense
facilitation, or resist adversarial attacks. To overcome these challenges, we
propose Slot, an advanced APT detection approach based on provenance graphs and
graph reinforcement learning. Slot excels in uncovering multi-level hidden
relationships, such as causal, contextual, and indirect connections, among
system behaviors through provenance graph mining. By pioneering the integration
of graph reinforcement learning, Slot dynamically adapts to new user activities
and evolving attack strategies, enhancing its resilience against adversarial
attacks. Additionally, Slot automatically constructs the attack chain according
to detected attacks with clustering algorithms, providing precise
identification of attack paths and facilitating the development of defense
strategies. Evaluations with real-world datasets demonstrate Slot's outstanding
accuracy, efficiency, adaptability, and robustness in APT detection, with most
metrics surpassing state-of-the-art methods. Additionally, case studies
conducted to assess Slot's effectiveness in supporting APT defense further
establish it as a practical and reliable tool for cybersecurity protection.


http://arxiv.org/abs/2410.17922
Guide for Defense (G4D): Dynamic Guidance for Robust and Balanced Defense in Large Language Models. (9%)
He Cao; Weidi Luo; Yu Wang; Zijing Liu; Bing Feng; Yuan Yao; Yu Li
  With the extensive deployment of Large Language Models (LLMs), ensuring their
safety has become increasingly critical. However, existing defense methods
often struggle with two key issues: (i) inadequate defense capabilities,
particularly in domain-specific scenarios like chemistry, where a lack of
specialized knowledge can lead to the generation of harmful responses to
malicious queries. (ii) over-defensiveness, which compromises the general
utility and responsiveness of LLMs. To mitigate these issues, we introduce a
multi-agents-based defense framework, Guide for Defense (G4D), which leverages
accurate external information to provide an unbiased summary of user intentions
and analytically grounded safety response guidance. Extensive experiments on
popular jailbreak attacks and benign datasets show that our G4D can enhance
LLM's robustness against jailbreak attacks on general and domain-specific
scenarios without compromising the model's general functionality.


http://arxiv.org/abs/2410.18210
Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attacks. (2%)
Samuele Poppi; Zheng-Xin Yong; Yifei He; Bobbie Chern; Han Zhao; Aobo Yang; Jianfeng Chi
  Recent advancements in Large Language Models (LLMs) have sparked widespread
concerns about their safety. Recent work demonstrates that safety alignment of
LLMs can be easily removed by fine-tuning with a few adversarially chosen
instruction-following examples, i.e., fine-tuning attacks. We take a further
step to understand fine-tuning attacks in multilingual LLMs. We first discover
cross-lingual generalization of fine-tuning attacks: using a few adversarially
chosen instruction-following examples in one language, multilingual LLMs can
also be easily compromised (e.g., multilingual LLMs fail to refuse harmful
prompts in other languages). Motivated by this finding, we hypothesize that
safety-related information is language-agnostic and propose a new method termed
Safety Information Localization (SIL) to identify the safety-related
information in the model parameter space. Through SIL, we validate this
hypothesis and find that only changing 20% of weight parameters in fine-tuning
attacks can break safety alignment across all languages. Furthermore, we
provide evidence to the alternative pathways hypothesis for why freezing
safety-related parameters does not prevent fine-tuning attacks, and we
demonstrate that our attack vector can still jailbreak LLMs adapted to new
languages.


http://arxiv.org/abs/2410.18312
Countering Autonomous Cyber Threats. (2%)
Kade M. Heckel; Adrian Weller
  With the capability to write convincing and fluent natural language and
generate code, Foundation Models present dual-use concerns broadly and within
the cyber domain specifically. Generative AI has already begun to impact
cyberspace through a broad illicit marketplace for assisting malware
development and social engineering attacks through hundreds of
malicious-AI-as-a-services tools. More alarming is that recent research has
shown the potential for these advanced models to inform or independently
execute offensive cyberspace operations. However, these previous investigations
primarily focused on the threats posed by proprietary models due to the until
recent lack of strong open-weight model and additionally leave the impacts of
network defenses or potential countermeasures unexplored. Critically,
understanding the aptitude of downloadable models to function as offensive
cyber agents is vital given that they are far more difficult to govern and
prevent their misuse. As such, this work evaluates several state-of-the-art FMs
on their ability to compromise machines in an isolated network and investigates
defensive mechanisms to defeat such AI-powered attacks. Using target machines
from a commercial provider, the most recently released downloadable models are
found to be on par with a leading proprietary model at conducting simple cyber
attacks with common hacking tools against known vulnerabilities. To mitigate
such LLM-powered threats, defensive prompt injection (DPI) payloads for
disrupting the malicious cyber agent's workflow are demonstrated to be
effective. From these results, the implications for AI safety and governance
with respect to cybersecurity is analyzed.


http://arxiv.org/abs/2410.17628
Is Smoothness the Key to Robustness? A Comparison of Attention and Convolution Models Using a Novel Metric. (1%)
Baiyuan Chen
  Robustness is a critical aspect of machine learning models. Existing
robustness evaluation approaches often lack theoretical generality or rely
heavily on empirical assessments, limiting insights into the structural factors
contributing to robustness. Moreo